aboutsummaryrefslogtreecommitdiffstats
path: root/pkg/covermerger
Commit message (Collapse)AuthorAgeFilesLines
* pkg/coveragedb: update file to subsystem info periodicallyTaras Madan2025-08-071-3/+3
| | | | | | #6070 explains the problem of data propagation. 1. Add weekly /cron/update_coverdb_subsystems. 2. Stop updating subsystems from coverage receiver API.
* all/mocks: regenerate with mockery v3Taras Madan2025-07-011-42/+48
|
* pkg/covermerger: read gcs files sequentiallyTaras Madan2025-04-042-42/+187
| | | | | | | Quarter long aggregation means thousands of gzip files. Opening all the files in parallel we struggle from: 1. Memory overhead. 2. GCS API errors. It can't read Attrs for 1500+ files.
* pkg/gcs: simplify interface, remove proxy typeTaras Madan2025-04-021-5/+1
|
* pkg/covermerger: skip record without target lineTaras Madan2025-03-281-0/+5
| | | | | | Some records have only information about the function covered. Let's skip them. ~2% of the records in BigQuery don't have information about line. It is a side effect of #5850.
* all/mocks: updateTaras Madan2025-03-281-0/+44
|
* all: use mockery config instead of go:generateTaras Madan2025-03-281-2/+0
|
* pkg/gcs: define Client interfaceTaras Madan2025-03-251-1/+1
| | | | | | Some functions are not the struct members now. Some functions deleted. Client mock generated.
* dashboard/app: support gerrit servers as a webgit sourceTaras Madan2025-02-131-6/+24
| | | | | Gerrit servers return base64 encoded file content. To get the raw data, &format=TEXT request param is needed.
* all: delete dead codeTaras Madan2025-02-101-9/+0
| | | | | go install golang.org/x/tools/cmd/deadcode@latest deadcode -test ./...
* go.mod: update mockeryTaras Madan2025-02-071-1/+1
|
* pkg/covermerger: group by func_nameTaras Madan2025-01-291-1/+1
| | | | It fixes the query error.
* pkg/covermerger: read function names from raw dataTaras Madan2025-01-291-1/+1
|
* pkg/coveragedb: store information about covered file functions in dbTaras Madan2025-01-291-2/+2
|
* pkg/covermerger: stream information about covered functions to dashboardTaras Madan2025-01-292-7/+81
|
* dashboard/app: test coverage /file linkTaras Madan2025-01-275-10/+76
| | | | | | | | 1. Init coveragedb client once and propagate it through context to enable mocking. 2. Always init coverage handlers. It simplifies testing. 3. Read webGit and coveragedb client from ctx to make it mockable. 4. Use int for file line number and int64 for merged coverage. 5. Add tests.
* dashboard/app: show manager unique coverageTaras Madan2025-01-272-5/+5
| | | | | | | | | | | | 1. Make heatmap testable, move out the spanner client instantiation. 2. Generate spannerdb.ReadOnlyTransaction mocks. 3. Generate spannerdb.RowIterator mocks. 4. Generate spannerdb.Row mocks. 5. Prepare spannerdb fixture. 6. Fixed html control name + value. 7. Added multiple tests. 8. Show line coverage from selected manager. 9. Propagate coverage url params to file coverage url.
* pkg/covermerger: more testsTaras Madan2024-12-221-0/+100
|
* pkg/covermerger: propagate context cancellationTaras Madan2024-12-192-18/+32
| | | | | | | | The problem is the deadlock happening on GCS storage error. GCS client establishes the connection when it has enough data to write. It is approximately 16M. The error happens on io.Writer access in the middle of the merge. This error terminates errgroup goroutine. Other goroutines remain blocked. This commit propagates termination signal to other goroutines.
* pkg/covermerger: remove excessive logsTaras Madan2024-12-191-1/+0
|
* pkg/coveragedb: test SaveMergeResultTaras Madan2024-12-192-29/+184
| | | | | | | | 1. Make interface testable. 2. Add Spanner interfaces. 3. Generate mocks for proxy interfaces. 4. Test SaveMergeResult. 5. Test MergeCSVWriteJSONL and coveragedb.SaveMergeResult integration.
* tools/syz-covermerger: upload coverage as jsonlTaras Madan2024-12-192-26/+29
| | | | | | | | | | | | | | | | | Previous implementation store only the summary of processed records. The summary was <1GB and single processing node was able to manipulate the data. Current implementation stores all the details about records read to make post-processing more flexible. This change was needed to get access to the source manager name and will help to analyze other details. This new implementation requires 20GB mem to process single day records. CSV log interning experiment allowed to merge using 10G. Quarter data aggregation will cost ~100 times more. The alternative is to use stream processing. We can process data kernel-file-by-file. It allows to /15000 memory consumption. This approach is implemented here. We're batching coverage signals by file and store per-file results in GCS JSONL file. See https://jsonlines.org/ to learn about jsonl.
* pkg/covermerger: always store detailsTaras Madan2024-12-163-26/+90
| | | | | Storing all the details about coverage data source we're able to better explain the origin. This origin data is currently used to get "manager" name.
* pkg/covermerger: aggregate data per-managerTaras Madan2024-12-123-7/+13
| | | | It enables us to see the manager unique coverage.
* dashboard/app: make WebGitURIProvider configurableTaras Madan2024-10-251-2/+2
| | | | | We don't want to scale the github repo usage. Let's eventually switch to the internal service.
* pkg/validator: target value is not empty by defaultTaras Madan2024-09-301-1/+1
| | | | | | The consistency make the code more predictable. We want all the values to be non-empty. If "empty" is an option - use validator.AnyOk(validator.EmptyStr(target), otherCheck(target)).
* dashboard/app: add file coverage pageTaras Madan2024-09-161-9/+19
| | | | | | | | | | | | | | It directly uses the coverage signals from BigQuery. There is no need to wait for the coverage_batch cron jobs. Looks good for debugging. Limitations: 1. It is slow. I know how to speed up but want to stabilize the UI first. 2. It is expensive because of the direct BQ requests. Limited to admin only because of it. 3. It merges only the commits reachable on github because of the gitweb throttling. After the UI stabilization I'll save all the required artifacts to spanner and make this page publicly available. To merge all the commits, not the github reachable only, http git caching instance is needed.
* pkg/covermerger: delete unused, simplifyTaras Madan2024-09-124-16/+15
|
* pkg/covermerger: delete trivial functionTaras Madan2024-08-291-7/+2
|
* pkg/covermerger: rename variables, delete unusedTaras Madan2024-08-294-46/+44
| | | | | Not all variables were renames after RepoBranchCommit refactoring. Some unused variables found.
* pkg/covermerger: optimize checkoutsTaras Madan2024-08-291-0/+9
| | | | | Every commit checkout takes >3 seconds. Let's optimize this operation to save on large merges.
* pkg/covermerger: use filepath.Join instead of plusTaras Madan2024-08-292-2/+3
|
* pkg/covermerger: we don't need branch infoTaras Madan2024-08-295-44/+31
| | | | | | | The first versions of this code used branches for git checkout. Later the git repos were merged and branch info usage was reduced. The latest changes switched the code from branch checkout to commit checkout. There is no need in branch info anymore and I don't see any use-cases for it.
* pkg/covermerger: clone commits not branchesTaras Madan2024-08-271-29/+25
| | | | | | | | | | | | Closes https://github.com/google/syzkaller/issues/5213. Some commits may be detached upstream. Specifying the commits we increase the cloning probability. Tested by: go run ./tools/syz-covermerger/ -workdir ../workdir-cover-aggregation/ -repo git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git -branch master -commit 3d5f968a177d468cd13568ef901c5be84d83d32b -namespace upstream -duration 7 -date-to 2024-08-23 -file-path-prefix drivers/clk/actions
* pkg/covermerger: never give up diffing the filesTaras Madan2024-08-231-0/+1
| | | | | | It hopefully fixes https://github.com/google/syzkaller/issues/4815. The default 1 second timeout is not what we want. To reproduce the problem, set timeout to 10 ms.
* pkg/validator: initial codeTaras Madan2024-08-221-9/+7
|
* pkg/covermerger: allow to use namespaces with dashTaras Madan2024-08-211-1/+1
| | | | | Current statement doesn't work for table names like "something-71". The namespace was backticked here before the code migration to golang.
* pkg/covermerger: fix bugTaras Madan2024-08-161-1/+1
|
* tools/syz-cover: filter by source commitTaras Madan2024-08-161-3/+7
|
* pkg/covermerger: store line detailsTaras Madan2024-08-162-11/+19
|
* pkg/covermerger: stop panic, do errorsTaras Madan2024-08-161-16/+26
|
* pkg/covermerger: refactor FileRecord, move parsing to makeRecordTaras Madan2024-08-163-66/+50
|
* pkg/covermerger: fix merge bugTaras Madan2024-08-162-2/+2
|
* tools/syz-covermerger: move problems logging closer to the guilty codeTaras Madan2024-08-142-5/+5
|
* tools/syz-covermerger: use pkg/logTaras Madan2024-08-142-4/+4
|
* tools/syz-cover: default commit is latestTaras Madan2024-08-141-1/+4
|
* tools/syz-cover: add toolFileCoverTaras Madan2024-08-141-0/+11
|
* all: fix EnableStorageReadClient errors processingTaras Madan2024-08-141-1/+3
|
* tools/syz-bq.sh: rewrite logic in golangTaras Madan2024-08-131-0/+124
|
* covermerger: add web-git source providerTaras Madan2024-08-093-3/+68
| | | | It allows to speed up the single file calculations because we don't need git clone.
generated by ggit (джигит) mrf-deployment (git 2.46.0) at 2026-03-11 19:50:23 +0000