1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
|
# syz-testbed
`syz-testbed` is the tool that simplifies the process of evaluating the
performance of different syzkaller versions (or configurations) against each
other. The tool automates checking out syzkaller repos, building them, running
`syz-manager`s and collecting/summarizing their results.
## Configuring syz-testbed
`syz-testbed` requires a JSON config file. An example of such a file:
```json
{
"workdir": "/tmp/syz-testbed-workdir/",
"corpus": "/tmp/corpus.db",
"target": "syz-manager",
"max_instances": 5,
"run_time": "24h",
"http": "0.0.0.0:50000",
"checkouts": [
{
"name": "first",
"repo": "https://github.com/google/syzkaller.git",
},
{
"name": "second",
"repo": "https://github.com/google/syzkaller.git",
"branch": "some-dev-branch",
}
],
"manager_config": {
"target": "linux/amd64",
"kernel_obj": "/tmp/linux-stable",
"image": "/tmp/kernel-image/bullseye.img",
"sshkey": "/tmp/kernel-image/bullseye.id_rsa",
"procs": 8,
"type": "qemu",
"vm": {
"count": 2,
"kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
"cpu": 2,
"mem": 2048
}
}
}
```
Given such a configuration file, `syz-testbed` will do the following:
1. Check out the `master` branch of `https://github.com/google/syzkaller.git`
into `/tmp/syz-testbed-workdir/checkouts/first/` and build it.
2. Check out the `some-dev-branch` of `https://github.com/google/syzkaller.git`
into `/tmp/syz-testbed-workdir/checkouts/second/` and build it.
3. Set up and run 3 instances of `first` and 2 instances of `second`
(`max_instances = 5`).
The directory structure looks as follows:
```
/tmp/syz-testbed-workdir/
└── checkouts
├── first
│ ├── run-first-0
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
│ ├── run-first-1
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
│ └── run-first-4
│ │ ├── log.txt
│ │ ├── manager.cfg
│ │ └── workdir
└── second
├── run-second-2
│ ├── log.txt
│ ├── manager.cfg
│ └── workdir
└── run-second-3
├── log.txt
├── manager.cfg
└── workdir
```
4. After 24 hours (as `run_hours` is 24), stop those 5 instances.
5. Create and run 2 instances of `first` and 3 instances of `second`.
6. <Repeat those steps over and over>
The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM
signal. Also, if one of the instances has exited due to some error, this also
stops the whole experiment.
## Web interface
The tool has a simple web interface that displays the current information about
the experiment (the number of active and finished instances, the time until
instances are stopped, etc.) and the latest statistics collected from the
`syz-manager`s.
If the `benchmp` parameter points to the `syz-benchcmp` executable, then the web
interface can also generate graphs of various parameters over time or the number
of executions.
In order to enable the interface, set the `http` parameter to the IP address and
port to which `syz-testbed` should bind. E.g. `"http": "0.0.0.0:50000"`.
## Statistics
`syz-testbed` provides two "views" of the statistics:
1. `complete` - only includes data from the finished instances (i.e. those that
have been running for `run_hours`).
2. `all` - also includes the data from the currently active instances. The
statistics from the finished instances is winded back to match the current
uptime of the active instances.
Therefore, the statistics is laid out the following way.
```bash
$ tree -L 2 /tmp/syz-testbed-workdir/
/tmp/syz-testbed-workdir/
├── stats_all
│ ├── benches
│ │ ├── avg_first.txt
│ │ ├── avg_second.txt
│ ├── bugs.csv
│ ├── checkout_stats.csv
│ └── instance_stats.csv
├── stats_completed
│ ├── benches
│ │ ├── avg_first.txt
│ │ ├── avg_second.txt
│ ├── bugs.csv
│ ├── checkout_stats.csv
│ └── instance_stats.csv
└── testbed.csv
```
1. `bugs.csv` contains all the bugs found by the running instances. If a single
checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a
union of bugs found by them. The purpose is ultimately to collect all bugs
that could be found by that version of syzkaller.
2. Statistics that is generated by individual `syz-manager`s is saved into
`instance_stats.csv`. The same data is also averaged among instances that
belong to the same checkouts and saved into the `checkout_stats.csv` file.
3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a
single checkout are averaged and saved into the corresponding files in the
`benches` folder.
The statics is updated once every 90 seconds.
## Running syz-testbed
First, checkout the most recent version of syzkaller itself:
```
$ git clone https://github.com/google/syzkaller.git
```
Then, build `syz-testbed`:
```
$ cd syzkaller/tools/syz-testbed/
$ go build
```
Write and save the configuration file (e.g. into the `config.json` file). Then,
`syz-testbed` can be run using the following command:
```
$ ./syz-testbed -config config.json
```
Stopping the `syz-testbed` process results in stopping all the syzkaller instances.
## Testing syz-repro
`syz-testbed` can also be used to test syzkaller's ability to reproduce bugs. To do
so, set the `target` property in the `syz-testbed`'s config file to `syz-repro`.
One can also specify the source of the crash log files. This is either just a folder,
whose files will be treated accordingly or it can be a syzkaller's workdir.
`input_logs` must point to the folde with crash logs - `syz-testbed` will traverse
it and treat each file as an input. Otherwise, `input_workdir` must be used.
For example:
```json
"repro_config": {
"input_workdir": "/tmp/some-syzkaller-workdir",
"crashes_per_bug": 2,
"skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"]
},
```
In this case, `syz-testbed` will traverse all bugs found by the syzkaller, skip
those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then
pick 2 random crash logs for each such bug for later processing.
`syz-testbed` will check out and compile the specified syzkaller instances and will
go on executing their `syz-repro`s on each picked up crash log file, as long as
the tool is not stopped.
|
