# syz-testbed `syz-testbed` is the tool that simplifies the process of evaluating the performance of different syzkaller versions (or configurations) against each other. The tool automates checking out syzkaller repos, building them, running `syz-manager`s and collecting/summarizing their results. ## Configuring syz-testbed `syz-testbed` requires a JSON config file. An example of such a file: ```json { "workdir": "/tmp/syz-testbed-workdir/", "corpus": "/tmp/corpus.db", "target": "syz-manager", "max_instances": 5, "run_time": "24h", "http": "0.0.0.0:50000", "checkouts": [ { "name": "first", "repo": "https://github.com/google/syzkaller.git", }, { "name": "second", "repo": "https://github.com/google/syzkaller.git", "branch": "some-dev-branch", } ], "manager_config": { "target": "linux/amd64", "kernel_obj": "/tmp/linux-stable", "image": "/tmp/kernel-image/trixie.img", "sshkey": "/tmp/kernel-image/trixie.id_rsa", "procs": 8, "type": "qemu", "vm": { "count": 2, "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage", "cpu": 2, "mem": 2048 } } } ``` Given such a configuration file, `syz-testbed` will do the following: 1. Check out the `master` branch of `https://github.com/google/syzkaller.git` into `/tmp/syz-testbed-workdir/checkouts/first/` and build it. 2. Check out the `some-dev-branch` of `https://github.com/google/syzkaller.git` into `/tmp/syz-testbed-workdir/checkouts/second/` and build it. 3. Set up and run 3 instances of `first` and 2 instances of `second` (`max_instances = 5`). The directory structure looks as follows: ``` /tmp/syz-testbed-workdir/ └── checkouts    ├── first    │   ├── run-first-0    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    │   ├── run-first-1    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    │   └── run-first-4    │   │   ├── log.txt    │   │   ├── manager.cfg    │   │   └── workdir    └── second       ├── run-second-2       │   ├── log.txt       │   ├── manager.cfg       │   └── workdir       └── run-second-3          ├── log.txt          ├── manager.cfg          └── workdir ``` 4. After 24 hours (as `run_hours` is 24), stop those 5 instances. 5. Create and run 2 instances of `first` and 3 instances of `second`. 6. The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM signal. Also, if one of the instances has exited due to some error, this also stops the whole experiment. ## Web interface The tool has a simple web interface that displays the current information about the experiment (the number of active and finished instances, the time until instances are stopped, etc.) and the latest statistics collected from the `syz-manager`s. If the `benchmp` parameter points to the `syz-benchcmp` executable, then the web interface can also generate graphs of various parameters over time or the number of executions. In order to enable the interface, set the `http` parameter to the IP address and port to which `syz-testbed` should bind. E.g. `"http": "0.0.0.0:50000"`. ## Statistics `syz-testbed` provides two "views" of the statistics: 1. `complete` - only includes data from the finished instances (i.e. those that have been running for `run_hours`). 2. `all` - also includes the data from the currently active instances. The statistics from the finished instances is winded back to match the current uptime of the active instances. Therefore, the statistics is laid out the following way. ```bash $ tree -L 2 /tmp/syz-testbed-workdir/ /tmp/syz-testbed-workdir/ ├── stats_all │   ├── benches │   │   ├── avg_first.txt │   │   ├── avg_second.txt │   ├── bugs.csv │   ├── checkout_stats.csv │   └── instance_stats.csv ├── stats_completed │   ├── benches │   │   ├── avg_first.txt │   │   ├── avg_second.txt │   ├── bugs.csv │   ├── checkout_stats.csv │   └── instance_stats.csv └── testbed.csv ``` 1. `bugs.csv` contains all the bugs found by the running instances. If a single checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a union of bugs found by them. The purpose is ultimately to collect all bugs that could be found by that version of syzkaller. 2. Statistics that is generated by individual `syz-manager`s is saved into `instance_stats.csv`. The same data is also averaged among instances that belong to the same checkouts and saved into the `checkout_stats.csv` file. 3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a single checkout are averaged and saved into the corresponding files in the `benches` folder. The statics is updated once every 90 seconds. ## Running syz-testbed First, checkout the most recent version of syzkaller itself: ``` $ git clone https://github.com/google/syzkaller.git ``` Then, build `syz-testbed`: ``` $ cd syzkaller/tools/syz-testbed/ $ go build ``` Write and save the configuration file (e.g. into the `config.json` file). Then, `syz-testbed` can be run using the following command: ``` $ ./syz-testbed -config config.json ``` Stopping the `syz-testbed` process results in stopping all the syzkaller instances. ## Testing syz-repro `syz-testbed` can also be used to test syzkaller's ability to reproduce bugs. To do so, set the `target` property in the `syz-testbed`'s config file to `syz-repro`. One can also specify the source of the crash log files. This is either just a folder, whose files will be treated accordingly or it can be a syzkaller's workdir. `input_logs` must point to the folde with crash logs - `syz-testbed` will traverse it and treat each file as an input. Otherwise, `input_workdir` must be used. For example: ```json "repro_config": { "input_workdir": "/tmp/some-syzkaller-workdir", "crashes_per_bug": 2, "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"] }, ``` In this case, `syz-testbed` will traverse all bugs found by the syzkaller, skip those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then pick 2 random crash logs for each such bug for later processing. `syz-testbed` will check out and compile the specified syzkaller instances and will go on executing their `syz-repro`s on each picked up crash log file, as long as the tool is not stopped.