docs: add section on descriptions testing

The sys/OS/test/* tests are currently mostly undocumented.
Add initial documentation for these tests.

The "External Dependencies" section is just moved with no changes.

2 files changed, 74 insertions, 14 deletions

diff --git a/docs/pseudo_syscalls.md b/docs/pseudo_syscalls.md
index 0fb991806..82c2bfeb8 100644
--- a/docs/pseudo_syscalls.md
+++ b/docs/pseudo_syscalls.md
@@ -72,20 +72,6 @@ are violated (e.g. passing `NULL` to a `non-NULL` argument, or passing
 `-1` as file descriptor) and produce errors/warnings. `volatile` prevents
 that.
 
-The implementation must not use any external libraries nor external headers,
-except for the most basic and standard ones (like `<unistd.h>` and
-`<sys/mman.h>`). In particular, it must not depend on libraries/headers
-installed by additional packages nor on headers for recently added kernel
-subsystems. External dependencies have proved to be brittle and easily cause
-build breakage because all dependencies will be required for any build/run on
-the fuzzer and any C reproducer. For example, packages/headers may be missing
-on some distros, named differently, be of a wrong version, broken, or conflict
-with other headers. Unfortunately, there is no way to reliably specify such
-dependencies and requirements for C programs. Therefore, if the pseudo-syscall
-requires definitions of some structures, constants, or helper functions, these
-should be described in the executor code itself as minimally as possible (they
-will be part of C reproducers).
-
 Now, to handle the pseudo-syscall properly we have to update the
 `isSupportedSyzkall` in
 [syscalls_linux.go](../pkg/host/syscalls_linux.go) and add a particular
@@ -103,3 +89,36 @@ Finally, run `make generate`. Now you can use it in a syscall
 description file as if it was a regular system call:
 
     syz_mycall(arg0 pid, arg1 const[0])
+
+<div id="dependencies"/>
+
+## External Dependencies
+
+The implementation must not use any external libraries nor external headers,
+except for the most basic and standard ones (like `<unistd.h>` and
+`<sys/mman.h>`). In particular, it must not depend on libraries/headers
+installed by additional packages nor on headers for recently added kernel
+subsystems. External dependencies have proved to be brittle and easily cause
+build breakage because all dependencies will be required for any build/run on
+the fuzzer and any C reproducer. For example, packages/headers may be missing
+on some distros, named differently, be of a wrong version, broken, or conflict
+with other headers. Unfortunately, there is no way to reliably specify such
+dependencies and requirements for C programs. Therefore, if the pseudo-syscall
+requires definitions of some structures, constants, or helper functions, these
+should be described in the executor code itself as minimally as possible (they
+will be part of C reproducers).
+
+## Testing
+
+Each new pseudo-syscall should have at least one test in `sys/OS/test`.
+See [Linux tests](/sys/linux/test) for an example. A tests is just a program
+with checked syscall return values. There should be at least one test
+that contains "the main successful scenario" of using the pseudo-syscall.
+See [io_uring test](/sys/linux/test/io_uring) as a good example.
+Such tests are important because they ensure that the pseudo-syscall code
+does not contain "stupid" bugs (e.g. crash on NULL-deref each time),
+that it is possible for the fuzzer to come up with the successful scenario
+(as a combination of the pseudo-syscall and the surrounding descriptions)
+and that it will continue to work in future.
+See [Testing of descriptions](syscall_descriptions.md#testing)
+for details about the tests.
diff --git a/docs/syscall_descriptions.md b/docs/syscall_descriptions.md
index c3f00c5f4..d3f8f5d17 100644
--- a/docs/syscall_descriptions.md
+++ b/docs/syscall_descriptions.md
@@ -252,3 +252,44 @@ remember to add `CROSS_COMPILE=arm-linux-gnueabi-/aarch64-linux-gnu-/powerpc64le
 If the kernel was built into a separate directory (with `make O=output_dir`, remember to put .config
 into output_dir, this will be helpful if you'd like to work on different arch at the same time)
 then also set `$LINUXBLD` to the location of the build directory.
+
+<div id="testing"/>
+
+### Testing of descriptions
+
+Descriptions themselves may contain bugs. After running `syz-manager` with the new descriptions
+it's always useful to check the kernel code coverage report available in the `syz-manager` web UI.
+The report allows to assess if everything one expects to be covered is in fact covered,
+and if not, where the fuzzer gets stuck. However, this is a useful but quite indirect assessment
+of the descriptions correctness. The fuzzer may get around some bugs in the descriptions by diverging
+from what the descriptions say, but it makes it considerably harder for the fuzzer to progress.
+
+Tests stored in `sys/OS/test/*` provide a more direct testing of the descriptions. Each test is just
+a program with checked syscall return values. The syntax of the programs is not currently documented,
+but look at the [existing examples](/sys/linux/test) and at the program [deserialization code](/prog/encoding.go).
+`AUTO` keyword can be used as a value for consts and pointers, for pointers it will lead to
+some reasonable sequential allocation of memory addresses.
+
+It's always good to add a test at least for "the main successful scenario" for the subsystem.
+It will ensure that the descriptions are actually correct and that it's possible for the fuzzer
+to come up with the successful scenario. See [io_uring test](/sys/linux/test/io_uring) as a good example.
+
+The tests can be run with the `syz-runtest` utility as:
+```
+make runtest && bin/syz-runtest -config manager.config
+```
+`syz-runtest` boots multiple VMs and runs these tests in different execution modes inside of the VMs.
+
+However, full `syz-runtest` run takes time, so while developing the test, it's more handy to run it
+using the `syz-execprog` utility. To run the test, copy `syz-execprog`, `syz-executor` and the test
+into a manually booted VM and then run the following command inside of the VM:
+```
+syz-execprog -debug -threaded=0 mytest
+```
+It will show results of all executed syscalls. It's also handy for manual debugging of pseudo-syscall code:
+if you add some temporal `debug` calls to the pseudo-syscall, `syz-execprog -debug` will show their output.
+
+The test syntax can be checked by running:
+```
+go test -run=TestSysTests ./pkg/csource
+```