pkg/bisect: mark jobs with untestable history as failed

Background info:

When a bisection is started, we only know the kernel commit on which
syzkaller encountered the crash. Before the actual bisection begins,
bisect() needs to find a good commit. Then we can bisect between them.

For fix bisections bisect() tests HEAD. If HEAD doesn't have the bug
the fixing commit must be somewhere in the middle. For cause bisection
we test a number of old releases, starting with the newest release.

Both this commit range search and the actual bisection later use test()
to build the kernel and run the reproducer.

During actual bisections we invoke test() for every step. If any test()
invocation returns a non nil error, the bisection it was called from
is aborted. Any non-fatal errors should be signaled via the testResult
returned from test(). For this reason a build/boot failure does not
return an error. Instead testResult.verdict will be vcs.BisectSkip.

The Problem:

Given the following call stack:
bisect() -> commitRange() -> commitRangeFor{Fix,Cause}() -> test()

Previously we reported fix bisections, where HEAD was build broken and
cause bisections where all tested releases were build broken as
inconclusive.

This is confusing for users. For fix bisections it looks like the
fixing commit is either the commit from the original crash or HEAD.
For cause bisections it looks like the breaking commit is either the
original commit or the commit of the oldest tested release. Neither
is correct.

For fix bisections we see this, when the HEAD of a tested branch is
build broken. When this happens all attempted fix bisections will get
nonsense results. For cause bisections we see this, when changes to the
bisection compilers or test rootfs cause us to not be able to build or
boot very old kernels.

These inconclusive bisection results will not be retried automatically
and we don't have an easy way to clear them.

The Solution:

For fix bisections: Retry the bisection later. If HEAD is completely
broken we will know, because fuzzing will stop.

For cause bisections: Mark the bisection as failed. The result is
unlikely to change in the future without intervention by the syzbot
admins. Users won't bother looking at those bisections and the
dashboard already has code to mass-retry failed bisections.

- In test(): Populate testResult.rep with a meaningful report before
  returning, after build/boot failures.
- In bisect(): Explicitly check the testResult.verdict of the last
  commit tested in commitRange(), instead of using testResult.rep==nil
  as an oracle for aborting the bisection.
- In bisect(): Don't return an inconclusive result when build/boot
  failures prevent us from finding a commit range to bisect between.

0 files changed, 0 insertions, 0 deletions