1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
|
## Reproduce a bug with syzbot's downloadable assets
As a part of every bug report, syzbot shares downloadable assets -- that is,
disk images and kernel binaries on which the bug was originally found.
This document serves as a guide on how to use those assets to reproce such bugs
locally.
### A sample report
To be more specific, let's take this syzbot report: [[syzbot] [hfs?] kernel BUG
in hfsplus_bnode_put](https://lore.kernel.org/all/000000000000efee7905fe4c9a46@google.com/).
```
syzbot has found a reproducer for the following issue on:
HEAD commit: 40f71e7cd3c6 Merge tag 'net-6.4-rc7' of git://git.kernel.o..
git tree: upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=10482ae3280000
kernel config: https://syzkaller.appspot.com/x/.config?x=7ff8f87c7ab0e04e
dashboard link: https://syzkaller.appspot.com/bug?extid=005d2a9ecd9fbf525f6a
compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xz
kernel image: https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gz
```
There are 4 linked assets:
* The bootable VM disk image on which the bug was found: `https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz`
* **The image is suitable both for GCE and for qemu**.
* The `vmlinux` file that can be used e.g. for report symbolization or for `gdb`-based debugging: `https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xz`
* The separate `bzImage` file (it is already included in the disk image): `https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xz`
* The filesystem image that is mounted in the reproducer: `https://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gz`
All these links are also reachable from the web dashboard.
#### Run a C reproducer
Boot a VM:
```
$ wget 'https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz'
$ unxz disk-40f71e7c.raw.xz
$ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=./disk-40f71e7c.raw,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1
```
Build and run the C reproducer:
```
$ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000'
$ gcc repro.c -lpthread -static -o repro
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro root@127.0.0.1:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 'chmod +x ./repro && ./repro'
```
Wait a minute and notice a crash report in the qemu's serial output:
```
[ 91.956238][ T81] ------------[ cut here ]------------
[ 91.957508][ T81] kernel BUG at fs/hfsplus/bnode.c:618!
[ 91.958645][ T81] invalid opcode: 0000 [#1] PREEMPT SMP KASAN
[ 91.959861][ T81] CPU: 0 PID: 81 Comm: kworker/u5:3 Not tainted 6.4.0-rc6-syzkaller-00195-g40f71e7cd3c6 #0
```
#### Run a syz reproducer directly
For some bugs, there's either no C reproducer or it's not reliable enough. In
that case, `syz` reproducers might be useful.
You'll need to [check out and build](/docs/linux/setup.md#go-and-syzkaller)
syzkaller first. The fastest way to do it is as follows (assuming Docker is
installed and configured on your machine):
```
$ git clone https://github.com/google/syzkaller.git
$ cd syzkaller
$ ./tools/syz-env make
```
Then boot a VM exactly like in the previous section.
Download and run the syz reproducer:
```
$ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000'
$ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./bin/linux_amd64/* ./repro.syz root@127.0.0.1:/root/
$ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz'
```
In some time, you'll see the same bug report in the VM's serial output.
The commands above execute the `./syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz`
command inside the VM. More details can be found in [this document](/docs/reproducing_crashes.md).
#### Use the `tools/syz-crush` tool
The `syz-crush` automatizes the steps above: it sets up and boots a pool of VMs
and runs the given `C` or `syz` reproducer in them.
First, download the disk image and reproducers (see instructions above).
Then, go to the syzkaller checkout and build the `syz-crush` tool:
```
$ make crush
```
Prepare a config file (let it be `config.json`):
```
{
"name": "test",
"http": "0.0.0.0:0",
"target": "linux/amd64",
"image": "/tmp/disk-40f71e7c.raw",
"syzkaller": "/tmp/syzkaller",
"workdir": "/tmp/syzkaller/workdir",
"type": "qemu",
"procs": 6,
"vm": {
"count": 5,
"cmdline": "root=/dev/sda1",
"cpu": 2,
"mem": 2048,
"qemu_args": "-machine pc-q35-7.1 -enable-kvm"
}
}
```
You need to replace `/tmp/syzkaller` with the location of your syzkaller
checkout and `/tmp/disk-40f71e7c.raw` with the location of the bootable disk
image.
Run the tool:
```
$ mkdir workdir
$ ./bin/syz-crush -config config.json repro.syz
```
#### Reproducing ARM64 bugs
If the bug was found on an ARM64 instance (e.g. the manager name is
`ci-upstream-gce-arm64`), you may use the following qemu command as a reference:
```
$ qemu-system-aarch64 -machine virt -cpu cortex-a57 -smp 4 -m 4G -nographic -drive file=disk.raw,if=none,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -kernel Image -net user,hostfwd=tcp::10023-:22 -net nic -append "root=/dev/vda2" -accel tcg,thread=multi
```
### Problems
#### The bug doesn't reproduce
If the `C` reproder did not work, try to run the `syz` reproducer.
If there's still no success, it might be that relatively rare case when the
execution environment becomes important. Syzbot fuzzes kernels on GCE VMs, which
might have a different instruction set / execution speed than locally run qemu
VMs. These changes might be critical for the generated reproducer.
There's unfortunately no universal solution.
Note that you can always ask syzbot to
[apply your git patch and re-run the reproducer](/docs/syzbot.md#testing-patches).
It will be run in the same GCE environment where the bug was originally found.
See also [this document](/docs/syzbot.md#crash-does-not-reproduce).
#### Assets are not downloadable
The downloadable assets are not stored infinitely. Syzbot keeps them until the
bug is fixed or marked as invalid + 30 days after that.
So if you cannot download the assets using the links from the email, this might
be a sign that the bug is actually no longer worth looking at.
#### Qemu doesn't boot
A [recent qemu problem](https://lore.kernel.org/qemu-devel/da39abab9785aea2a2e7652ed6403b6268aeb31f.camel@linux.ibm.com/)
may prevent it from booting large kernel images. Add `-machine pc-q35-7.1` to
the qemu args to make it work.
|
