From 6e2b10b1a483820e4126ff63b684dafd8689e994 Mon Sep 17 00:00:00 2001 From: Dmitry Vyukov Date: Sat, 25 Jul 2020 09:42:05 +0200 Subject: docs: add section on descriptions testing The sys/OS/test/* tests are currently mostly undocumented. Add initial documentation for these tests. The "External Dependencies" section is just moved with no changes. --- docs/pseudo_syscalls.md | 47 +++++++++++++++++++++++++++++++------------- docs/syscall_descriptions.md | 41 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 14 deletions(-) (limited to 'docs') diff --git a/docs/pseudo_syscalls.md b/docs/pseudo_syscalls.md index 0fb991806..82c2bfeb8 100644 --- a/docs/pseudo_syscalls.md +++ b/docs/pseudo_syscalls.md @@ -72,20 +72,6 @@ are violated (e.g. passing `NULL` to a `non-NULL` argument, or passing `-1` as file descriptor) and produce errors/warnings. `volatile` prevents that. -The implementation must not use any external libraries nor external headers, -except for the most basic and standard ones (like `` and -``). In particular, it must not depend on libraries/headers -installed by additional packages nor on headers for recently added kernel -subsystems. External dependencies have proved to be brittle and easily cause -build breakage because all dependencies will be required for any build/run on -the fuzzer and any C reproducer. For example, packages/headers may be missing -on some distros, named differently, be of a wrong version, broken, or conflict -with other headers. Unfortunately, there is no way to reliably specify such -dependencies and requirements for C programs. Therefore, if the pseudo-syscall -requires definitions of some structures, constants, or helper functions, these -should be described in the executor code itself as minimally as possible (they -will be part of C reproducers). - Now, to handle the pseudo-syscall properly we have to update the `isSupportedSyzkall` in [syscalls_linux.go](../pkg/host/syscalls_linux.go) and add a particular @@ -103,3 +89,36 @@ Finally, run `make generate`. Now you can use it in a syscall description file as if it was a regular system call: syz_mycall(arg0 pid, arg1 const[0]) + +
+ +## External Dependencies + +The implementation must not use any external libraries nor external headers, +except for the most basic and standard ones (like `` and +``). In particular, it must not depend on libraries/headers +installed by additional packages nor on headers for recently added kernel +subsystems. External dependencies have proved to be brittle and easily cause +build breakage because all dependencies will be required for any build/run on +the fuzzer and any C reproducer. For example, packages/headers may be missing +on some distros, named differently, be of a wrong version, broken, or conflict +with other headers. Unfortunately, there is no way to reliably specify such +dependencies and requirements for C programs. Therefore, if the pseudo-syscall +requires definitions of some structures, constants, or helper functions, these +should be described in the executor code itself as minimally as possible (they +will be part of C reproducers). + +## Testing + +Each new pseudo-syscall should have at least one test in `sys/OS/test`. +See [Linux tests](/sys/linux/test) for an example. A tests is just a program +with checked syscall return values. There should be at least one test +that contains "the main successful scenario" of using the pseudo-syscall. +See [io_uring test](/sys/linux/test/io_uring) as a good example. +Such tests are important because they ensure that the pseudo-syscall code +does not contain "stupid" bugs (e.g. crash on NULL-deref each time), +that it is possible for the fuzzer to come up with the successful scenario +(as a combination of the pseudo-syscall and the surrounding descriptions) +and that it will continue to work in future. +See [Testing of descriptions](syscall_descriptions.md#testing) +for details about the tests. diff --git a/docs/syscall_descriptions.md b/docs/syscall_descriptions.md index c3f00c5f4..d3f8f5d17 100644 --- a/docs/syscall_descriptions.md +++ b/docs/syscall_descriptions.md @@ -252,3 +252,44 @@ remember to add `CROSS_COMPILE=arm-linux-gnueabi-/aarch64-linux-gnu-/powerpc64le If the kernel was built into a separate directory (with `make O=output_dir`, remember to put .config into output_dir, this will be helpful if you'd like to work on different arch at the same time) then also set `$LINUXBLD` to the location of the build directory. + +
+ +### Testing of descriptions + +Descriptions themselves may contain bugs. After running `syz-manager` with the new descriptions +it's always useful to check the kernel code coverage report available in the `syz-manager` web UI. +The report allows to assess if everything one expects to be covered is in fact covered, +and if not, where the fuzzer gets stuck. However, this is a useful but quite indirect assessment +of the descriptions correctness. The fuzzer may get around some bugs in the descriptions by diverging +from what the descriptions say, but it makes it considerably harder for the fuzzer to progress. + +Tests stored in `sys/OS/test/*` provide a more direct testing of the descriptions. Each test is just +a program with checked syscall return values. The syntax of the programs is not currently documented, +but look at the [existing examples](/sys/linux/test) and at the program [deserialization code](/prog/encoding.go). +`AUTO` keyword can be used as a value for consts and pointers, for pointers it will lead to +some reasonable sequential allocation of memory addresses. + +It's always good to add a test at least for "the main successful scenario" for the subsystem. +It will ensure that the descriptions are actually correct and that it's possible for the fuzzer +to come up with the successful scenario. See [io_uring test](/sys/linux/test/io_uring) as a good example. + +The tests can be run with the `syz-runtest` utility as: +``` +make runtest && bin/syz-runtest -config manager.config +``` +`syz-runtest` boots multiple VMs and runs these tests in different execution modes inside of the VMs. + +However, full `syz-runtest` run takes time, so while developing the test, it's more handy to run it +using the `syz-execprog` utility. To run the test, copy `syz-execprog`, `syz-executor` and the test +into a manually booted VM and then run the following command inside of the VM: +``` +syz-execprog -debug -threaded=0 mytest +``` +It will show results of all executed syscalls. It's also handy for manual debugging of pseudo-syscall code: +if you add some temporal `debug` calls to the pseudo-syscall, `syz-execprog -debug` will show their output. + +The test syntax can be checked by running: +``` +go test -run=TestSysTests ./pkg/csource +``` -- cgit mrf-deployment