docs: update USB fuzzing documentation

1 files changed, 234 insertions, 22 deletions

diff --git a/docs/linux/external_fuzzing_usb.md b/docs/linux/external_fuzzing_usb.md
index f9da23ced..2a842446d 100644
--- a/docs/linux/external_fuzzing_usb.md
+++ b/docs/linux/external_fuzzing_usb.md
@@ -1,48 +1,260 @@
 External USB fuzzing for Linux kernel
 =====================================
 
-This page describes the current state of external USB fuzzing support in syzkaller.
-Note, that it's still in development and things might change.
+Syzkaller support fuzzing the Linux kernel USB subsystem externally
+(as it would be done by plugging in a physical USB device with e.g. [Facedancer](https://github.com/usb-tools/Facedancer)).
+This allowed to find over [100 bugs](/docs/linux/found_bugs_usb.md) in the Linux kernel USB stack so far.
+This is still in development and things might change.
+
+USB fuzzing consists of 3 parts:
+
+1. Syzkaller changes that are now upstream.
+2. Kernel interface for USB device emulation, which can be found [here](https://github.com/google/kasan/commits/usb-fuzzer).
+3. KCOV changes that allow to collect coverage from background threads and interrupts
+(the former can be found [here](https://github.com/google/kasan/commits/usb-fuzzer), the latter in still in development).
+
+Currently syzkaller defines 5 USB syzcalls (see [this](/sys/linux/vusb.txt) and [this](/executor/common_usb.h)):
 
-Some details about how it works can be found:
+1. `syz_usb_connect` - connects a USB device.
+2. `syz_usb_disconnect` - disconnects a USB device.
+3. `syz_usb_control_io` - sends or receives a control message over endpoint 0.
+4. `syz_usb_ep_write` - sends a message to an endpoint.
+4. `syz_usb_ep_read` - receives a message from an endpoint.
 
-1. In the OffensiveCon 2019 [slides](https://docs.google.com/presentation/d/1z-giB9kom17Lk21YEjmceiNUVYeI6yIaG5_gZ3vKC-M/edit?usp=sharing).
+More details can be found:
+
+1. In the OffensiveCon 2019 "Coverage-Guided USB Fuzzing with Syzkaller" talk
+([slides](https://docs.google.com/presentation/d/1z-giB9kom17Lk21YEjmceiNUVYeI6yIaG5_gZ3vKC-M/edit?usp=sharing), [video](https://www.youtube.com/watch?v=1MD5JV6LfxA)).
 2. In [this](https://marc.info/?l=linux-usb&m=155551883403285&w=2) email.
 
-This allowed to find over [100 bugs](/docs/linux/found_bugs_usb.md) in the Linux kernel USB stack so far.
+A few major things that need to be done:
+
+1. Implement support for multiple interfaces per configuration (this is required to properly emulate some USB devices like CDC NCM).
+2. Collect coverage from interrupts (this is required to enable better fuzzing of USB drivers after enumeration completes).
+3. Add descriptions for all main USB classes.
+4. Upstream KCOV changes.
+5. Upstream the kernel interface for USB device emulation.
 
-How to set this up:
+Some ideas for things that can be done:
+
+1. Add a mode for standalone fuzzing of physical USB hosts (by using e.g. Raspberry Pi Zero, see below).
+This includes at least: a. making sure that current USB emulation implementation works properly on different OSes (there are some differences);
+and b. using USB requests coming from the host as a signal (like coverage) to enable "signal-driven" fuzzing.
+2. Generate syzkaller programs from usbmon trace that is produced by actual USB devices (this should make the fuzzer to go significantly deeper into the USB drivers code).
+
+Syzkaller descriptions for USB fuzzing can be found [here](/sys/linux/vusb.txt).
+
+
+## Setting up
 
 1. Checkout the `usb-fuzzer` branch from https://github.com/google/kasan
 
-2. Configure and build the kernel. You need to enable `CONFIG_USB_FUZZER=y`, `CONFIG_USB_DUMMY_HCD=y` and all the USB drivers you're interested in fuzzing:
+2. Configure the kernel (at the very least `CONFIG_USB_FUZZER=y` and `CONFIG_USB_DUMMY_HCD=y` need to be enabled).
+
+    The easiest option is to use the [config](/dashboard/config/upstream-usb.config) from the syzbot USB fuzzing instance.
+
+    Another option is to use the USB config generation [script](/dashboard/config/generate-config-usb.sh).
+    This script allows to extract enabled USB related configs from a set of existing `.config` files.
+    Right now it extracts configs only from [one](/dashboard/config/distros) of the Ubuntu kernel's configs.
 
-   ```
-   menu config -> Device Drivers -> USB Support ->
-     -> USB Gadget Support (enable) ->
-       -> USB Peripheral Controller -> Dummy HCD (enable)
-       -> USB Gadget Fuzzer (enable)
-   ```
+    ``` bash
+    cd ./dashboard/config/
+    # Put relevant .configs into ./distros/
+    CC=$COMPILER_BINARY_PATH SOURCEDIR=$KERNEL_SOURCE_PATH ./generate-config-usb.sh
+    ```
 
-3. Update syzkaller descriptions by extracting USB device info using the instructions below.
+3. Build the kernel.
 
-4. Enable `syz_usb_connect`, `syz_usb_disconnect`, `syz_usb_control_io` and `syz_usb_ep_write` syscalls in the manager config.
+4. Optionally update syzkaller descriptions by extracting USB IDs using the instructions below.
 
-5. Set `sandbox` to `none` in the manager config.
+5. Enable `syz_usb_connect`, `syz_usb_disconnect`, `syz_usb_control_io`, `syz_usb_ep_write` and `syz_usb_ep_read` syzcalls in the manager config.
 
-6. Pass `dummy_hcd.num=8` to the kernel command line in the maganer config.
+6. Set `sandbox` to `none` in the manager config.
 
-7. Run.
+7. Pass `dummy_hcd.num=8` to the kernel command line in the maganer config.
 
-Syzkaller descriptions for USB fuzzing can be found here: [1](/sys/linux/vusb.txt), [2](/sys/linux/init_vusb.go) and [3](/sys/linux/init_vusb_ids.go).
+8. Run.
 
 
-## Updating syzkaller USB descriptions
+## Updating syzkaller USB IDs
+
+Syzkaller uses a list of hardcoded [USB IDs](/sys/linux/init_vusb_ids.go) that are [patched](/sys/linux/init_vusb.go) into the `syz_usb_connect` syzcall by syzkaller runtime.
+One of the ways to make syzkaller target only particular USB drivers is to alter that list.
+The instructions below describe a way to generate syzkaller USB IDs for all USB drivers enabled in your .config.
 
 1. Apply [this](/tools/syz-usbgen/usb_ids.patch) kernel patch.
 
 2. Build and boot the kernel.
 
-3. Connect some USB device to it (e.g. with `syz-exeprog usb.log`, where `usb.log` is some program that utilizes the `syz_usb_connect` syzcall).
+3. Connect a USB HID device. In case you're using a `CONFIG_USB_FUZZER=y` kernel, use the provided [keyboard emulation program](/tools/syz-usbgen/keyboard.c).
+
+4. Use [syz-usbgen](/tools/syz-usbgen/usbgen.go) script to update [syzkaller descriptions](/sys/linux/init_vusb_ids.go):
+
+    ```
+    ./bin/syz-usbgen KERNEL_LOG ./sys/linux/init_vusb_ids.go
+    ```
+
+
+## Running reproducers with Raspberry Pi Zero W
+
+It's possible to run syzkaller USB reproducers by using a Linux board plugged into a physical USB host.
+These instructions describe how to set this up on a Raspberry Pi Zero W, but any other board that has a working USB UDC driver can be used as well.
+
+1. Download `raspbian-stretch-lite.img` from [here](https://www.raspberrypi.org/downloads/raspbian/).
+
+2. Flash the image into an SD card as described [here](https://www.raspberrypi.org/documentation/installation/installing-images/linux.md).
+
+3. Enable UART as described [here](https://www.raspberrypi.org/documentation/configuration/uart.md).
+
+4. Boot the board and get a shell over UART as described [here](https://learn.adafruit.com/raspberry-pi-zero-creation/give-it-life). You'll need a USB-UART module for that. The default login credentials are `pi` and `raspberry`.
+
+5. Get the board connected to the internet (plug in a USB Ethernet adapter or follow [this](https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md)).
+
+6. Update: `sudo apt-get update && sudo apt-get dist-upgrade && sudo rpi-update && sudo reboot`.
+
+7. Install useful packages: `sudo apt-get install vim git`.
+
+8. Download and install Go:
+
+    ``` bash
+    curl https://dl.google.com/go/go1.10.8.linux-armv6l.tar.gz -o go1.10.8.linux-armv6l.tar.gz
+    tar -xf go1.10.8.linux-armv6l.tar.gz
+    mv go goroot-1.10.8
+    mkdir gopath-1.10.8
+    export GOPATH=~/gopath-1.10.8
+    export GOROOT=~/goroot-1.10.8
+    export PATH=~/goroot-1.10.8/bin:$PATH
+    export PATH=~/gopath-1.10.8/bin:$PATH
+    ```
+
+9. Download syzkaller, apply the patch below and build `syz-executor`:
+
+    ``` c
+    diff --git a/executor/common_usb.h b/executor/common_usb.h
+    index e342d808..278c2f4e 100644
+    --- a/executor/common_usb.h
+    +++ b/executor/common_usb.h
+    @@ -269,9 +269,7 @@ static volatile long syz_usb_connect(volatile long a0, volatile long a1, volatil
+    
+            // TODO: consider creating two dummy_udc's per proc to increace the chance of
+            // triggering interaction between multiple USB devices within the same program.
+    -       char device[32];
+    -       sprintf(&device[0], "dummy_udc.%llu", procid);
+    -       rv = usb_fuzzer_init(fd, speed, "dummy_udc", &device[0]);
+    +       rv = usb_fuzzer_init(fd, speed, "20980000.usb", "20980000.usb");
+            if (rv < 0) {
+                    debug("syz_usb_connect: usb_fuzzer_init failed with %d\n", rv);
+                    return rv;
+    diff --git a/executor/executor.cc b/executor/executor.cc
+    index 34949a01..1afcb288 100644
+    --- a/executor/executor.cc
+    +++ b/executor/executor.cc
+    @@ -604,8 +604,8 @@ retry:
+                            call_extra_cover = true;
+                    }
+                    if (strncmp(syscalls[call_num].name, "syz_usb_connect", strlen("syz_usb_connect")) == 0) {
+    -                       prog_extra_timeout = 2000;
+    -                       call_extra_timeout = 2000;
+    +                       prog_extra_timeout = 5000;
+    +                       call_extra_timeout = 5000;
+                    }
+                    if (strncmp(syscalls[call_num].name, "syz_usb_control_io", strlen("syz_usb_control_io")) == 0)
+                            call_extra_timeout = 300;
+    ```
+
+    ``` bash
+    go get -u -d github.com/google/syzkaller/...
+    cd ~/gopath-1.10.8/src/github.com/google/syzkaller
+    # Put the patch above into ./syzkaller.patch
+    git apply ./syzkaller.patch
+    make executor
+    mkdir ~/syz-bin
+    cp bin/linux_arm/syz-executor ~/syz-bin/
+    ```
+
+10. Build `syz-execprog` on your host machine for arm32 with `make TARGETARCH=arm execprog` and copy to `~/syz-bin` onto the SD card.
+
+11. Make sure that ou can now execute syzkaller programs:
+
+    ``` bash
+    cat socket.log
+    r0 = socket$inet_tcp(0x2, 0x1, 0x0)
+    sudo ./syz-bin/syz-execprog -executor ./syz-bin/syz-executor -threaded=0 -collide=0 -procs=1 -nocgroups -nonetdev -nonetreset -notun -debug socket.log
+    ```
+
+12. Setup the dwc2 USB gadget driver:
+
+    ```
+    echo "dtoverlay=dwc2" | sudo tee -a /boot/config.txt
+    echo "dwc2" | sudo tee -a /etc/modules
+    sudo reboot
+    ```
+
+13. Get Linux kernel headers following [this](https://github.com/notro/rpi-source/wiki).
+
+14. Download the fuzzer module:
+
+    ``` bash
+    mkdir module
+    cd module
+    wget https://raw.githubusercontent.com/google/kasan/usb-fuzzer/drivers/usb/gadget/fuzzer/fuzzer.c
+    wget https://raw.githubusercontent.com/google/kasan/usb-fuzzer/include/uapi/linux/usb/fuzzer.h
+    ```
+
+    Apply the following change:
+
+    ``` c
+    diff --git a/fuzzer.c b/fuzzer.c
+    index 308c540..68d43b9 100644
+    --- a/fuzzer.c
+    +++ b/fuzzer.c
+    @@ -17,7 +17,7 @@
+     #include <linux/usb/gadgetfs.h>
+     #include <linux/usb/gadget.h>
+     
+    -#include <uapi/linux/usb/fuzzer.h>
+    +#include "fuzzer.h"
+     
+     #define        DRIVER_DESC "USB fuzzer"
+     #define DRIVER_NAME "usb-fuzzer-gadget"
+    ```
+
+    Add a `Makefile`:
+
+    ``` make
+    obj-m := fuzzer.o
+    KDIR := /lib/modules/$(shell uname -r)/build
+    PWD := $(shell pwd)
+    default:
+    	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
+    ```
+
+    And build with `make`.
+
+15. Insert the module with `sudo insmod fuzzer.ko`.
+
+16. Build and test the [keyboard emulator program](https://raw.githubusercontent.com/google/syzkaller/up-usb-docs/tools/syz-usbgen/keyboard.c):
+
+    ``` bash
+    # Connect the board to some USB host.
+    wget https://raw.githubusercontent.com/google/syzkaller/up-usb-docs/tools/syz-usbgen/keyboard.c
+    gcc keyboard.c -o keyboard
+    sudo ./keyboard
+    # Make sure you see the letter 'x' being entered on the host.
+    ```
+
+17. You should now be able to execute syzkaller USB programs:
+
+    ``` bash
+    $ cat usb.log
+    r0 = syz_usb_connect(0x0, 0x24, &(0x7f00000001c0)={{0x12, 0x1, 0x0, 0x8e, 0x32, 0xf7, 0x20, 0xaf0, 0xd257, 0x4e87, 0x0, 0x0, 0x0, 0x1, [{{0x9, 0x2, 0x12, 0x1, 0x0, 0x0, 0x0, 0x0, [{{0x9, 0x4, 0xf, 0x0, 0x0, 0xff, 0xa5, 0x2c}}]}}]}}, 0x0)
+    $ sudo ./syz-bin/syz-execprog -executor ./syz-bin/syz-executor -threaded=0 -collide=0 -procs=1 -nocgroups -nonetdev -nonetreset -notun -debug usb.log
+    ```
+
+18. Follow [this](https://www.raspberrypi.org/documentation/configuration/wireless/access-point.md) to setup Wi-Fi hotspot.
+
+19. Follow [this](https://www.raspberrypi.org/documentation/remote-access/ssh/) to enable ssh.
+
+20. Optionally solder [Zero Stem](https://zerostem.io/) onto your Raspberry Pi Zero W.
 
-4. Use [syz-usbgen](/tools/syz-usbgen/usbgen.go) script to update [syzkaller descriptions](/sys/linux/init_vusb_ids.go).
+21. You can now connect the board to an arbitrary USB port, wait for it to boot, join its Wi-Fi network, ssh onto it, and run arbitrary syzkaller USB programs.