docs: describe syz-testbed

1 files changed, 138 insertions, 0 deletions

diff --git a/docs/syz_testbed.md b/docs/syz_testbed.md
new file mode 100644
index 000000000..098ddd2e8
--- /dev/null
+++ b/docs/syz_testbed.md
@@ -0,0 +1,138 @@
+# syz-testbed
+
+`syz-testbed` is the tool that simplifies the process of evaluating the effect
+the performance of different syzkaller versions (or configurations) against each
+other. The tool automates checking out syzkaller repos, building them, running
+`syz-manager`s and collecting/summarizing their results.
+
+## Configuring syz-testbed
+
+`syz-testbed` takes a JSON config file as its input. An example of such a file:
+
+```json
+{
+  "workdir": "/tmp/syz-testbed-workdir/",
+  "corpus": "/tmp/corpus.db",
+  "checkouts": [
+    {
+      "name": "first",
+      "repo": "https://github.com/google/syzkaller.git",
+      "count": 2
+    },
+    {
+      "name": "second",
+      "repo": "https://github.com/google/syzkaller.git",
+      "branch": "some-dev-branch",
+      "count": 2
+    }
+  ],
+  "manager_config": {
+	  "target": "linux/amd64",
+	  "http": "0.0.0.0:50000",
+	  "kernel_obj": "/tmp/linux-stable",
+	  "image": "/tmp/kernel-image/stretch.img",
+	  "sshkey": "/tmp/kernel-image/stretch.id_rsa",
+	  "procs": 8,
+	  "type": "qemu",
+	  "vm": {
+          "count": 2,
+          "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
+          "cpu": 2,
+          "mem": 2048
+	  }
+  }
+}
+```
+
+When run with such a configuration file, `syz-testbench` will do the following:
+1. Check out the `master` branch of `https://github.com/google/syzkaller.git`
+   into `/tmp/workdir/checkouts/first/`.
+   2. Generate two independent config files for that syzkaller. They will have
+   separate `workdir`'s (`/tmp/syz-testbed-workdir/checkouts/first/workdir_1/`
+   and ``/tmp/syz-testbed-workdir/checkouts/first/workdir_2`), separate names
+   (`first-1` and `first-2`) and separate ports (50000 and
+   50001). `/tmp/corpus.db` is copied into each of the work directories and will
+   be used by `syz-manager`s as the initial corpus.
+3. Build syzkaller at `/tmp/syz-testbed-workdir/first/`.
+4. Check out the `some-dev-branch` of
+   `https://github.com/google/syzkaller.git` into
+   `/tmp/syz-testbed-workdir/second/`.
+5. Do the same as was done in the steps 2 and 3, but for the `second` folder.
+6. The resulting directory structure looks as follows
+
+```
+/tmp/syz-testbed-workdir/
+└── checkouts
+    ├── first
+<...>
+    │   ├── syz_1.cnf
+    │   ├── syz_2.cnf
+<...>
+    │   ├── workdir_1
+    │   └── workdir_2
+    └── second
+<...>
+        ├── syz_1.cnf
+        ├── syz_2.cnf
+<...>
+        ├── workdir_1
+        └── workdir_2
+```
+
+7. Finally `syz-testbed` runs all the `syz-manager` instances it has
+   prepared. The config file above results in 4 instances: 2 belonging the
+   `first` checkout and 2 belonging to the `second` one.
+
+Right after the 7th step `syz-testbed` begins to collect and aggregate the data
+from the running syzkaller instances.
+
+```bash
+$ tree -L 2 /tmp/syz-testbed-workdir/
+/tmp/syz-testbed-workdir/
+├── benches
+│   ├── avg_first.txt
+│   └── avg_second.txt
+├── bugs.csv
+├── checkouts
+│   ├── first
+│   └── second
+├── checkout_stats.csv
+└── instance_stats.csv
+```
+
+1. `bugs.csv` contains all the bugs found by the running instances. If a single
+   checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a
+   union of bugs found by them. The purpose is ultimately to collect all bugs
+   that could be found by that version of syzkaller.
+2. Statistics that is generated by individual `syz-manager`s is saved into
+   `instance_stats.csv`. The same data is also averaged among instances that
+   belong to the same checkouts and saved into the `checkout_stats.csv` file.
+3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a
+   single checkout are averaged and saved into the corresponding files in the
+   `benches` folder.
+
+The statics is updated once every 90 seconds.
+
+## Running syz-testbed
+
+First, checkout the most recent version of syzkaller itself:
+
+```
+$ git clone https://github.com/google/syzkaller.git
+```
+
+Then, build `syz-testbed`:
+
+```
+$ cd syzkaller/tools/syz-testbed/
+$ go build
+```
+
+Write and save the configuration file (e.g. into the `config.json` file). Then,
+`syz-testbed` can be run using the following command:
+
+```
+$ ./syz-testbed -config config.json
+```
+
+Stopping the `syz-testbed` process results in stopping all the syzkaller instances.