aboutsummaryrefslogtreecommitdiffstats
path: root/pkg/bisect/bisect.go
Commit message (Collapse)AuthorAgeFilesLines
* all: use any instead of interface{}Dmitry Vyukov2025-12-221-1/+1
| | | | Any is the preferred over interface{} now in Go.
* pkg/osutil: move Semaphore from pkg/instanceDmitry Vyukov2025-11-241-2/+2
| | | | | | | | Semaphore is a very low-level primitive type, while pkg/instance is a very high-level package with lots of deps. Semaphore does not belong there, and may lead to cyclic deps if we use it more. Move it to pkg/osutil. It's not really OS-specific, but we don't have a better package.
* pkg/osutil: make VerboseError nest other errorsAleksandr Nogikh2025-10-011-1/+1
| | | | | After this change it fits more naturally into the Go's error functionality.
* all: apply linter auto fixesTaras Madan2025-07-171-2/+3
| | | | ./tools/syz-env bin/golangci-lint run ./... --fix
* all: use min/max functionsDmitry Vyukov2025-01-171-14/+2
| | | | They are shorter, more readable, and don't require temp vars.
* pkg/bisect: ignore irrelevant lost connection crashesAleksandr Nogikh2024-12-031-5/+17
| | | | | | | These have been the cause of too many invalid bisection results recently. This is the first step towards the more universal approach of #5414.
* pkg/bisect: recognize lost connection errorsAleksandr Nogikh2024-12-031-0/+5
| | | | Only select them as relevant if there are not other crash types.
* pkg/vcs: change HeadCommit to CommitDmitry Vyukov2024-10-151-3/+3
| | | | | | | Currently we have HeadCommit function that returns info about the HEAD commit. Change it to a more flexible Commit function that can return info about any commit. This will be used in future changes.
* pkg/build: use the build environment in clean() callsFlorent Revest2024-10-141-17/+19
| | | | | | This unifies the build() and clean() interfaces such that if a custom compiler or make binary is provided in the manager or bisection config, they can be taken into account by the clean() interface.
* pkg/build/linux: support building with a custom make binaryFlorent Revest2024-10-141-0/+2
| | | | | | Certain environments might need a specific make command or wrap make calls with extra logic. This lets users provide a path to a custom make binary.
* pkg/build: handle OOM-killed build errorSabyrzhan Tasbolatov2024-09-231-11/+3
| | | | | | | Handle SIGKILL (exit code = 137) on osutil.Run() during Linux kernel image building and return build.InfraError without reporting. Fixes: https://github.com/google/syzkaller/issues/5317
* all: follow new linter recommendationsTaras Madan2024-09-101-53/+57
|
* all: rename build_jobs to build_cpusAleksandr Nogikh2024-08-221-2/+2
| | | | The latter is a better suitable name.
* syz-ci: accept a BuildJobs parameterAleksandr Nogikh2024-08-221-0/+2
| | | | | The parameter defines the maximum number of CPUs involved in the kernel build process.
* pkg/bisect: set a lower bound for BisectBad verdictAleksandr Nogikh2024-07-151-1/+2
| | | | | | | | | The "1 crashed, 9 OK" cases are a frequent reason of invalid bisection results on syzbot. Let's define a cutoff for a BisectBad verdict and use it to prevent such obvious outliers. We cannot safely declare such results as BisectGood either, so let's return BisectSkip in this case.
* pkg/bisect: make bisection more robustSimone Weiß2024-03-181-1/+5
| | | | | | | Bisection should not fail if the Kconfig or the baseline config have issues. Broken kernel sources might lead to issues when parsing Kconfig, ignore this and proceed with the original config. If the baseline config is not parseable, proceed anyway as this is an optional parameter to begin with.
* pkg/bisect: validate log() argumentsAleksandr Nogikh2023-09-081-1/+4
| | | | | | | | I couldn't manage to add "pkg/bisect.env.log" to "govet"."settings"."printf"."funcs", so let's use another way suggested by govet documentation. Fix one instance of invalid arguments.
* pkg/bisect: test a subset of releasesAleksandr Nogikh2023-08-251-1/+57
| | | | | | | | | | For older bugs (or for bugs on stable trees), our cause bisection strategy times out while trying to iterate over all reachable tags. Try to be smarter and only take a subset of them, thus limiting the time we spend detecting the bug-free release. Closes #3376.
* pkg/bisect: test merge base for cross-tree bisectionsAleksandr Nogikh2023-08-241-4/+25
| | | | | | | | | | | | | | | | | Consider the following situation: we're fix bisecting an lts bug on the mainline tree. Reproducer works on lts HEAD, but does not crash the mainline. If the bug was not introduced in the mainline, but it's a purely lts bug, bisection would currently end up pointing to the mainline HEAD (in #4123, we assume env.commit to be a commit on the tree we're currently bisecting, otherwise we'd always hit the same merge base commit). Let's first determine 100% correct bisection range on the mainline tree by testing that both the HEAD does not crash (we already do) and by testing that the new starting commit (=merge base) can be crashed by the reproducer.
* pkg/bisect: start cause bisections from any commitsAleksandr Nogikh2023-08-221-0/+16
| | | | | | | | For cause bisections, don't require Kernel.Commit to be reachable from Kernel.Branch. We can start cause bisections from any existing commit. This should help with linux-next bisections, where no older HEADs are ever reachable from each new `master`.
* pkg/bisect: consider bug's existenceAleksandr Nogikh2023-08-111-6/+97
| | | | | | | | | | | | If there's a merge from a branch that was based on a much older revision, it's likely that it does not YET contain the bug. If we don't consider this possibility, we're likely to get invalid fix bisection results for bugs that existed only for a short period of time. For every fix bisection step, determine first whether the guilty revision is reachable from it. Cache resuls to speed up processing. See #4117.
* pkg/bisect: refactor bisect.goAleksandr Nogikh2023-08-111-24/+29
| | | | | Move predicate code out of env.bisect(). The function is already too complicated.
* pkg/bisect: don't log verdicts twiceAleksandr Nogikh2023-08-081-1/+0
| | | | | We incorrectly log crash verdicts twice, which makes bisection logs more confusing. Don't do that.
* all: use errors.As instead of .(type)Taras Madan2023-07-241-23/+28
|
* all: use special placeholder for errorsTaras Madan2023-07-241-3/+3
|
* syz-ci: specify per-manager bisection backportsAleksandr Nogikh2023-07-201-1/+6
| | | | | | | | It might be the case that the kernels that are being fuzzed on syz-ci require their own backports to build/test older revisions during bisection. Let users specify it in the syz-ci config.
* pkg/bisect: estimate confidence in the resultAleksandr Nogikh2023-07-141-13/+75
| | | | | | | | | | | | | | | | Estimate reproducer's flakiness more carefully, as it can help us get better results. During config minimization, do not let reproducibility drop too low. For each test() run, estimate our confidence in the result. For now, only consider the false negative case: if we got no crashes, it's likely that the bug is there, but the reproducer was not lucky enough. Given an estimate of reproduction likelihood, we can easily calculate the chance of an invalid result: (1-probability)^runs. Combine all individual test() confidences into Result.Confidence. If the resulting Confidence is too low, ignore the bisection result.
* pkg/bisect: fix SYZFATAL bisectionsAleksandr Nogikh2023-07-071-1/+2
| | | | | After the previous change, pkg/bisect is unable to bisect SYZFATAL errors. Fix the bug in the logic and add a test.
* pkg/report: ignore transient crashesAleksandr Nogikh2023-07-061-2/+17
| | | | | | | | | There are cases when a crash, even though it didn't occur during boot and image testing phases, still should not be counted during bisection. Otherwise we risk diverting the whole process in the wrong direction. One of such problem classes is getting SYZFATAL errors when we are not bisecting a SYZFATAL crash.
* pkg/bisect: try to drop instrumentationAleksandr Nogikh2023-07-051-10/+13
| | | | | | | | | | Refactor Minimize(). As a part of Minimize(), attempt to drop all unnecessary instrumentation and test whether the kernel is still crashing. This should reduce the number of irrelevant crashes during bisection and thus increase bisection result quality.
* pkg/bisect: remember the most frequent report typesAleksandr Nogikh2023-07-051-3/+77
| | | | | | | | | This will later help in adjusting kernel configuration. Also, adjust how test() determines the crash report: pick the one with the most frequent type. Earlier we were taking the last crash report, but this seems to have a bias towards the crashes that take longer to appear (e.g. rcu stalls).
* pkg/bisect: change verdict calculation rulesAleksandr Nogikh2023-06-291-20/+40
| | | | | | | | | 1) For the good verdict, demand a certain minimum number of good runs to be present. 2) For the bad verdict, demand that at least 50% of runs did not fail with boot/image test error. Refactor the code and add tests.
* pkg/bisect: support bisections on other treesAleksandr Nogikh2023-06-291-1/+24
| | | | | | | | | | | | | | | | | | | The current code only supports fix/cause bisections when the known bad commit is reachable from Kernel.Repo/Kernel.Branch. Add a CrossTree parameter to pkg/bisect. If it's set to true and we're doing a fix bisection, the bisection algorithm first operates with the original commit message (i.e. checks that it indeed crashes the kernel and performs config minimization), but the actual bisection starts from the merge base of Commit and Branch. We could have calculated the merge base outside of pkg/bisect and just started the algorithm from that merge base, but there's a problem: there's no guarantee that the kernel will build/boot with a syzbot config at the merge base. So we take the commit known to work well and then assume that the bug is also present on the merge base commit. If it were not present, we wouldn't have found a fix commit from Branch anyway.
* pkg/bisect: skip commits with too many boot/test errorsAleksandr Nogikh2023-06-151-2/+5
| | | | | | | | | If there are only a few non-crashed results among many boot/test errors, we cannot really be sure that the commit is good. It might be that the reproducer is not 100% reliable and we just needed more runs. Require > 50% of runs to be successful in order to claim that the revision is good. Otherwise skip it.
* pkg/bisect: treat repro checkout errors as infra errorsAleksandr Nogikh2023-05-091-1/+1
| | | | This will let us later restart them automatically.
* pkg/bisect: abort on infrastructure errorsAleksandr Nogikh2023-05-091-12/+30
| | | | | | There's not much sense to continue the bisection if something in the testing infrastructure got broken. Indicate it with the InfraError error.
* pkg/bisect: mark jobs with untestable history as failedSpace Meyer2023-05-051-36/+73
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Background info: When a bisection is started, we only know the kernel commit on which syzkaller encountered the crash. Before the actual bisection begins, bisect() needs to find a good commit. Then we can bisect between them. For fix bisections bisect() tests HEAD. If HEAD doesn't have the bug the fixing commit must be somewhere in the middle. For cause bisection we test a number of old releases, starting with the newest release. Both this commit range search and the actual bisection later use test() to build the kernel and run the reproducer. During actual bisections we invoke test() for every step. If any test() invocation returns a non nil error, the bisection it was called from is aborted. Any non-fatal errors should be signaled via the testResult returned from test(). For this reason a build/boot failure does not return an error. Instead testResult.verdict will be vcs.BisectSkip. The Problem: Given the following call stack: bisect() -> commitRange() -> commitRangeFor{Fix,Cause}() -> test() Previously we reported fix bisections, where HEAD was build broken and cause bisections where all tested releases were build broken as inconclusive. This is confusing for users. For fix bisections it looks like the fixing commit is either the commit from the original crash or HEAD. For cause bisections it looks like the breaking commit is either the original commit or the commit of the oldest tested release. Neither is correct. For fix bisections we see this, when the HEAD of a tested branch is build broken. When this happens all attempted fix bisections will get nonsense results. For cause bisections we see this, when changes to the bisection compilers or test rootfs cause us to not be able to build or boot very old kernels. These inconclusive bisection results will not be retried automatically and we don't have an easy way to clear them. The Solution: For fix bisections: Retry the bisection later. If HEAD is completely broken we will know, because fuzzing will stop. For cause bisections: Mark the bisection as failed. The result is unlikely to change in the future without intervention by the syzbot admins. Users won't bother looking at those bisections and the dashboard already has code to mass-retry failed bisections. - In test(): Populate testResult.rep with a meaningful report before returning, after build/boot failures. - In bisect(): Explicitly check the testResult.verdict of the last commit tested in commitRange(), instead of using testResult.rep==nil as an oracle for aborting the bisection. - In bisect(): Don't return an inconclusive result when build/boot failures prevent us from finding a commit range to bisect between.
* syz-ci: gate concurrent env.Test executionsAleksandr Nogikh2023-01-191-1/+2
| | | | This will help reduce the number of overcommitted VMs.
* syz-ci: move build semaphore closer to buildsAleksandr Nogikh2023-01-191-1/+2
| | | | | | | | Currently it's held during the whole job processing, which can take too long. Adjust it so that it's only taken when we really begin to build the kernel or syzkaller.
* pkg/instance: move BuildKernel() args to structSpace Meyer2023-01-091-2/+9
|
* syz-ci/jobs: use linker supplied in syz-ci configSpace Meyer2023-01-091-1/+2
| | | | | | Previously we only used the linter from the syz-ci config when building the kernel for regular fuzzing. We were missing some plumbing to have this setting reach patch testing and bisection jobs.
* pkg/bisect: use default compiler during bisection where possibleSpace Meyer2022-10-071-13/+15
| | | | | | | This allows us to bisect at least recently introduced bugs, where the manager that found the bug uses a non standard compiler. This is usefull during development of a new sanitizer for which a compiler with non-upstreamed patches is required.
* pkg/bisect: try to reidentify commit rebased after crashSpace Meyer2022-10-061-7/+51
| | | | | | | | | | | | | | When bisecting a breaking commit, syzkaller starts the bisection from the commit recorded in the last crash for the given bug. Previously the bisection was aborted should the commit no longer exist in the repo. Now we try to reidentify the breaking commit. For git pretty much the best we can do is to search a commit reachable from HEAD with the same title. Other VCS systems might have something better. Syzkaller will still first validate that the start commit is indeed broken in the way it expects. This prevents syzkaller from getting confused should we accidentally pick a completely unrelated commit.
* pkg/bisect: support clang for crash bisectionSpace Meyer2022-09-011-12/+13
| | | | | Before we hardcoded bisection to use gcc, now the compiler family can be configured in the bisection config.
* pkg/vcs: fetch linux upstream stable before bisectionsSpace Meyer2022-09-011-0/+5
|
* pkg/bisect: prevent crash in bisection after failed repo HEAD querySpace Meyer2022-09-011-2/+8
| | | | | | | | | | | | | | | | | | | During a bisection the call stack looks like this: Starting at env.bisect() in pkg/bisect/bisect.go bisect() -> Bisect() -> pred() -> test() -> build() Previously the `results[testRes1.com.Hash]` in pred() could run into a nil pointer deref and crash, when: * build() didn't bother to return current in certain error conditions * test() didn't handle cases where build() couldn't query current When test() returns an error, the bisection it was called from is aborted. Hence test() is expected to not simply bubble up recoverable errors, but instead return a testResult with verdict=vcs.BisectSkip. I ran into this with a linux branch, where syzkaller could not find any release tags, running into one of the cases where build() did not return current, even though it was able to obtain it.
* tools/syz-bisect: print hostname and bisection dateDenis Efremov2022-07-191-0/+6
| | | | | | | | Log bisection date and hostname of the bisecting machine for the debugging purposes. It makes more easy to check bisection problems on the bisecting host if something goes wrong. Signed-off-by: Denis Efremov <denis.e.efremov@oracle.com>
* syz-ci: remember syzkaller builds logsAleksandr Nogikh2022-06-281-16/+19
| | | | | This might help get more insight into patch testing failures, especially for old bugs.
* all: remember console output for all patch testsAleksandr Nogikh2022-06-081-3/+4
| | | | | | Currently syzbot only saves a log if there was a build/test error. Closes #3185
* all: capture compiler id during the build processAleksandr Nogikh2021-07-201-9/+8
| | | | | | | | | Default compilers are specified in the OS- and platform-dependent logic. It is more convenient to extract info about them during the kernel build itself, rather than during the manager object initialization. Apply the necessary changes throughout the code that is involved in building the kernels and processing information about this process.
generated by ggit (джигит) mrf-deployment (git 2.46.0) at 2026-03-12 00:27:35 +0000