| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
1. Make interface testable.
2. Add Spanner interfaces.
3. Generate mocks for proxy interfaces.
4. Test SaveMergeResult.
5. Test MergeCSVWriteJSONL and coveragedb.SaveMergeResult integration.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previous implementation store only the summary of processed records.
The summary was <1GB and single processing node was able to manipulate the data.
Current implementation stores all the details about records read to make post-processing more flexible.
This change was needed to get access to the source manager name and will help to analyze other details.
This new implementation requires 20GB mem to process single day records.
CSV log interning experiment allowed to merge using 10G.
Quarter data aggregation will cost ~100 times more.
The alternative is to use stream processing. We can process data kernel-file-by-file.
It allows to /15000 memory consumption.
This approach is implemented here.
We're batching coverage signals by file and store per-file results in GCS JSONL file.
See https://jsonlines.org/ to learn about jsonl.
|
| |
|
|
|
|
|
| |
I don't see any visible problems but the records in DB are not created.
Let's report the amount of records created at the end of the batch step.
+log the names of the managers
|
| |
|
|
| |
It enables us to see the manager unique coverage.
|
| | |
|
| |
|
|
|
|
| |
We currently merge bigquery data for every line coverage request.
Let's read cached lines coverage data from spanner instead.
It allows to get only 1 file version from git and skip the data merge step.
|
| |
|
|
|
| |
Instrumented lines + hit count gives more information than instrumented + covered lines.
Expected storage cost is at the same level.
|
| |
|
|
|
|
| |
It stores instrumented and covered lines numbers in the same table
where we store file coverage numbers.
These line numbers will be used to speed up the file coverage rendering.
|
| |
|
