aboutsummaryrefslogtreecommitdiffstats
path: root/sys/syz-sysgen/sysgen.go
Commit message (Collapse)AuthorAgeFilesLines
* sys/syz-sysgen: fix typomaoyixie2025-12-101-1/+1
|
* all: remove loop variables scopingTaras Madan2025-02-171-1/+0
|
* sys/syz-sysgen: serialize descriptions as gob and embedDmitry Vyukov2025-01-231-103/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of generating Go files with descriptions serialize them as gob and compress with flate. This significantly reduces build time, go vet time, and solves scalability problems with some static analysis tools. Reference times (all after rm -rf ~/.cache/go-build) before: TIME="%e %P %M" time go install ./syz-manager 48.29 577% 4824820 TIME="%e %P %M" time go test -c ./prog 56.28 380% 6973292 After: TIME="%e %P %M" time go install ./syz-manager 22.81 865% 859788 TIME="%e %P %M" time go test -c ./prog 12.74 565% 267760 syz-manager size before/after: 194712597 -> 83418407 -57% even provided we now embed all descriptions instead of just a single arch. Deflate/decoding time for a single Linux arch is ~330ms. Fixes #5542
* pkg/compiler: handle string syscall attributesFlorent Revest2024-12-091-0/+2
|
* sys/targets: mark big-endian targetsDmitry Vyukov2024-06-041-2/+2
| | | | | | Litte-endian is kind of default (except for s390). So instead of saying that each arch is litte-endian, mark only s390 as big-endian.
* executor: remove noshmem modeDmitry Vyukov2024-06-041-7/+2
| | | | | | | | | All OSes we have now support shmem. Support for Fuchia/Starnix/Windows wasn't implemented, but generally they support shared memory. Remove all of the complexity and code associated with noshmem mode. If/when we revive these OSes, it's easier to properly implement shmem mode for them.
* sys/syz-sysgen: remove all files in the case set of arches has changedDmitry Vyukov2024-06-041-0/+12
| | | | | If arched are renamed, old files will stay around and break builds. Remove all existing generated files for writing new ones.
* pkg/host: explicitly disable syz_execute_funcDmitry Vyukov2024-04-291-2/+5
| | | | | | | | | Disable the syscall in descriptions rather than in the code. This makes it more visible for users that it's disabled, and makes it less special (will not need to move this logic to host). Also change the condition in syz-sysgen to be more precise, otherwise syz_execute_func becomes unused function.
* sys/syz-sysgen: save flags's possible valuesFlorent Revest2024-02-011-5/+23
| | | | | This extends Target to contain a map of possible values for each int flag.
* sys/syz-sysgen: fail on undefined constsAleksandr Nogikh2023-10-041-1/+6
| | | | | | | | | | | | Ensure that every const mentioned in the description is defined for at least one architecture. We cannot do it inside pkg/compiler as it deals with only one arch at a time. Also, we cannot do it inside syz-check as it's not supposed to be called every time and may also include non-actionable warnings. So let's do it inside syz-sysgen, which is invoked all the time and has all the necessary data.
* sys: refactor const extractionAleksandr Nogikh2023-10-041-5/+10
| | | | | 1) Make FabricateSyscallConsts() operate on ConstFile. 2) Expose Pos inside ConstInfo.
* sys/targets: introduce HasCallNumber to reduce clutterGreg Steuck2023-04-251-4/+2
| | | | This centralizes all strings.HasPrefix(callName, "syz_") checks.
* all: ioutil is deprecated in go1.19 (#3718)Taras Madan2023-02-231-2/+1
|
* pkg/ast, pkg/compiler: support per-file metadataDmitry Vyukov2022-04-291-23/+2
| | | | | | | | | | | | | | | | | | | | | We have a bunch of hacks in syz-extract, syz-sysgen and syz-check with respect to description files unsupported on some arches, or that must not be part of make extract. Add 2 meta attribtues to files: meta noextract Tells `make extract` to not extract constants for this file. Though, `syz-extract` can still be invoked manually on this file. meta arches["arch1", "arch2"] Restricts this file only to the given set of architectures. `make extract` and ``make generate` will not use it on other architectures. Later we can potentially use meta attributes to specify git tree/commit that must be used for extraction. Maybe something else. Fixes #2754
* sys/syz-sysgen: generate possibly missing __NR/SYS definesAleksandr Nogikh2022-01-131-0/+25
| | | | | | | | | As the comiling machine may have a kernel version different from the tested one, not all definitions might be present. Generate sequences of ifndef in defs.h to avoid potential issues. Restrict __NR-related style checking rules to only checking common*.h files.
* sys/syz-sysgen: restore detection of unsupported kvm syscallsDmitry Vyukov2021-10-271-2/+13
| | | | | | | | The workaround for broken kvm kernel headers broke detection of unsupported syscalls. The diagnostic check checks that a syscall is unsupported on all arches, but arm/riscv kvm syscalls were not noted as unsupported (we dropped them before giving them to the compiler). As the result it looked like kvm syscalls are not unsupported on arm/riscv, and we did not produce the diagnostic.
* sys/syz-sysgen: refactor main functionDmitry Vyukov2021-10-271-44/+52
| | | | | | Split main function into 2 functions. It's too long and subsequent changes trigger lint warnings. No functional code changes.
* all: refactor fault injection into call propsAleksandr Nogikh2021-09-221-2/+2
| | | | | | | | | | | | Now that call properties mechanism is implemented, we can refactor fault injection. Unfortunately, it is impossible to remove all traces of the previous apprach. In reprolist and while performing syz-ci jobs, syzkaller still needs to parse the old format. Remove the old prog options-based approach whenever possible and replace it with the use of call properties.
* all: introduce call propertiesAleksandr Nogikh2021-09-221-0/+23
| | | | | | | | | Call properties let us specify how each individual call within a program must be executed. So far the only way to enforce extra rules was to pass extra program-level properties (e.g. that is how fault injection was done). However, it entangles the logic and not flexible enough. Implement an ability to pass properties along with each individual call.
* sys/syz-extract: restore kvm const extraction for arm64/ppc64Dmitry Vyukov2021-09-161-2/+1
| | | | | | | | | | | | Change #2755 disabled KVM for arm64/ppc64, but KVM is supported on these arches and has extensive support. It's pity to lose that support. The real root cause of the problem with arm64/ppc64 is that some severe compilation errors terminated compilation and did not let compiler spew all error messages. As the result we did not parse all of them and did not disable all of them. Re-try compilation multiple times instead of just 2 to fix this. Update #2754
* sys: skip kvm const extraction for non i386/amd64Aleksandr Nogikh2021-09-131-2/+2
| | | | | | | | | | | | It is impossible to compile a number of definitions in include/uapi/linux/kvm.h for other platforms, which leads to syz-extract failing to update constants. Skip processing of this file for all arches except i386 and amd64. This is a hacky and (hopefully) temporary solution until #2754 is implemented.
* pkg/host: fail loudly when coverage collection is enable w/o shmemPatrick Meyer2021-05-201-6/+6
| | | | Otherwise coverage collection just doesn't work.
* sys: commit empty source filesDmitry Vyukov2021-04-211-9/+0
| | | | | | Commit empty files into generated packages, so that the tree is buildable even w/o generated files and Go tools continue to work.
* all: use tool.Failf instead of local functionsDmitry Vyukov2020-12-251-9/+4
|
* pkg/cmdprof: merge into pkg/toolDmitry Vyukov2020-12-251-3/+2
| | | | cmdprof functionality seems to fit well into pkg/tool.
* sys/targets: add OS/Arch name constsDmitry Vyukov2020-10-261-2/+2
| | | | | | | | | | | | We use strings to identify OS/Arch. These strings are duplicated throughout the code base massively. golangci-lint points to possiblity of typos and duplication. We already had to define these names in pkg/csource and disable checking for prog package. A future change triggers such warnings in another package. Add OS/Arch name consts to sys/targets so that they can be used to refer to OS/Arch. Use the consts everywhere.
* pkg/compiler: merge const files into a single fileDmitry Vyukov2020-08-131-4/+5
| | | | | | | | | | | | | | | | | | | | We now have 8 arches for Linux and .const files produce lots of noise in PRs and lots of diffs. If 3 .txt files are touched, the PR will have 24 .const files, which will be intermixed with .txt files. Frequently const values are equal across arches, and even if they don't spreading a single value across 8 files is inconvinient. Merge all 8 *_arch.const files into a single .const file. See the test for details of the new format. The old format is still parsed for now, we can't update all OSes at once. For Linux this reduces number of const files/lines from 1288/96599 to 158/11603. Fixes #1983
* sys/syz-sysgen: ignore *_kvm.txt on linux/riscv64Tobias Klauser2020-06-261-3/+5
| | | | | | | The Linux RISC-V port in linux-next doesn't support KVM yet. Ignore it for now until KVM support is added upstream. Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
* sys/linux: first 64bit big-endian architecture s390xAlexander Egorenkov2020-06-251-7/+15
| | | | | | | | | | | | | | | * mmap syscall is special on Linux s390x because the parameters for this syscall are passed as a struct on user stack instead of registers. * Introduce the SyscallTrampolines table into targets.Target to address the above problem. * There is a bug in Linux kernel s390x which causes QEMU TCG to hang when KASAN is enabled. The bug has been fixed in the forthcoming Linux 5.8 version. Until then do not enable KASAN when using QEMU TCG, QEMU KVM shall have no problems with KASAN. Signed-off-by: Alexander Egorenkov <Alexander.Egorenkov@ibm.com>
* target: support of big-endian architecturesAlexander Egorenkov2020-06-231-2/+2
| | | | | | | | | | | | * Introduce the new target flag 'LittleEndian' which specifies of which endianness the target is. * Introduce the new requires flag 'littleendian' for tests to selectively enable/disable tests on either little-endian architectures or big-endian ones. * Disable KD unit test on s390x architecture because the test works only on little-endian architecture. Signed-off-by: Alexander Egorenkov <Alexander.Egorenkov@ibm.com>
* sys/syz-sysgen: be less verbose if no errors happenedDmitry Vyukov2020-06-191-4/+4
| | | | | | There is little point in printing all targets if no errors happened. Generation is fast, so this is not even working as a "progress bar". Only print target if there are any errors.
* sys/syz-sysgen: add -src and -out flagsMatthew Dempsky2020-06-021-7/+11
| | | | | | This is useful for integrating into Fuchsia's build system, where we need to be able to run syz-sysgen with a read-only source directory, and emit the output files elsewhere.
* sys/syz-sysgen: don't format descriptionsDmitry Vyukov2020-05-211-8/+2
| | | | | Formatting takes 10 seconds. make format will run go fmt anyway.
* sys/syz-sysgen: fix up linux/arm/kvm hackDmitry Vyukov2020-05-121-1/+1
| | | | We have some private files that need to be excluded as well... oh, hacks.
* sys/syz-extract: use -nostdinc on linuxDmitry Vyukov2020-05-111-2/+11
| | | | This makes the build completely hermetic.
* prog: refactor ANY to not fabricate new typesDmitry Vyukov2020-05-051-3/+3
| | | | | | | | | | Currently ANY implementation fabricates new types dynamically. This is something we don't do anywhere else, generally types come from compiler and all are static. Dynamic types will conflict with use of Ref in Arg optimization. Move ANY types creation into compiler. Update #1580
* prog: remove StructDescDmitry Vyukov2020-05-031-6/+2
| | | | | | | | | | | | Remove StructDesc, KeyedStruct, StructKey and all associated logic/complexity in prog and pkg/compiler. We can now handle recursion more generically with the Ref type, and Dir/FieldName are not a part of the type anymore. This makes StructType/UnionType simpler and more natural. Reduces size of sys/linux/gen/amd64.go from 5201321 to 4180861 (-20%). Update #1580
* pkg/cmdprof: add packageDmitry Vyukov2020-04-301-18/+2
| | | | | | | cmdprof simplifies cpu/memory profiling for command line tools. Use as: flag.Parse() defer cmdprof.Install
* pkg/compiler: deduplicate Types in descriptionsDmitry Vyukov2020-04-261-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add prog.Ref Type that serves as a proxy for real types and allows to deduplicate Types in generated descriptions. The Ref type is effectively an index in an array of types. Just before serialization pkg/compiler replaces real types with the Ref types and prepares corresponding array of real types. When a Target is registered in prog package, we do the opposite operation and replace Ref's with the corresponding real types. This brings improvements across the board: compiler memory consumption is reduced by 15%, test building time by 25%, descriptions size by 33%. Before: $ du -h sys/linux/gen 54M sys/linux/gen $ time GOMAXPROCS=1 go test -p=1 -c ./prog real 0m54.200s real 0m53.883s $ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog real 0m27.911s real 0m27.767s $ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen 20.59 100% 3200016 20.97 100% 3445976 20.25 100% 3209684 After: $ du -h sys/linux/gen 36M sys/linux/gen $ time GOMAXPROCS=1 go test -p=1 -c ./prog real 0m42.290s real 0m43.230s $ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog real 0m24.337s real 0m24.727s $ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen 19.11 100% 2764952 19.66 100% 2787624 19.35 100% 2749376 Update #1580
* sys/syz-sysgen: better formatting for syscall descriptionsDmitry Vyukov2020-04-191-3/+3
| | | | | We don't really need clang-format for auto-generated code. We can do better ourselves!
* prog: introduce call attributesDmitry Vyukov2020-04-191-11/+50
| | | | | | Add common infrastructure for syscall attributes. Add few attributes we want, but they are not implemented for now (don't affect behavior, this will follow).
* pkg/compiler: ensure consistency of syscall argument typesDmitry Vyukov2020-03-171-3/+15
| | | | | | | | | | | | | | | | | | Ensure that we don't have conflicting sizes for the same argument of the same syscall, e.g.: foo$1(a int16) foo$2(a int32) This is useful for several reasons: - we will be able avoid morphing syscalls into other syscalls - we will be able to figure out more precise sizes for args (lots of them are implicitly intptr, which is the largest type on most important arches) - found few bugs in linux descriptions Update #477 Update #502
* pkg/compiler: don't specify syscall consts for test OSDmitry Vyukov2020-03-171-0/+4
| | | | This is just tedious. Fabricate them on the fly.
* tools/syz-check: inject description compilation warningsDmitry Vyukov2019-12-201-1/+2
| | | | | | | | Currently we print them as part of `make genereate`, but nobody reads them, too much output each time. Don't print them in `make generate` and instead print in syz-check, the warn files are a good mechanism to handle "known warnings".
* .golangci.yml: add codeanalysis build tagDmitry Vyukov2019-05-151-0/+1
| | | | | | | | | Using a build tag to exclude files for golangci-lint reduces memory consumption (it does not parse them). The naive attempt with skip-dirs did not work. So add codeanalysis build tag and use it in auto-generated files. Update #977
* Makefile: don't compile all targets into target binariesDmitry Vyukov2018-08-021-5/+22
| | | | | | | | | | Currently target binaries contain support for all OS/arch combinations. However, obviously a fuchsia target binary won't test windows. For target binaries we need support only for a single target (with the exception of 386/arm target in amd64/arm64 binaries). So compile in only _the_ target into target binaries. This reduces akaros/amd64 fuzzer binary from 33 to 7 MB and execprog from 28 to 2 MB.
* executor: overhaulDmitry Vyukov2018-07-241-39/+74
| | | | | | | | | | | | | | | | | Make as much code as possible shared between all OSes. In particular main is now common across all OSes. Make more code shared between executor and csource (in particular, loop function and threaded execution logic). Also make loop and threaded logic shared across all OSes. Make more posix/unix code shared across OSes (e.g. signal handling, pthread creation, etc). Plus other changes along similar lines. Also support test OS in executor (based on portable posix) and add 4 arches that cover all execution modes (fork server/no fork server, shmem/no shmem). This change paves way for testing of executor code and allows to preserve consistency across OSes and executor/csource.
* executor: rework fallback coverageDmitry Vyukov2018-06-221-0/+6
| | | | | | | | | | We have fallback coverage implmentation for freebsd. 1. It's broken after some recent changes. 2. We need it for fuchsia, windows, akaros, linux too. 3. It's painful to work with C code. Move fallback coverage to ipc package, fix it and provide for all OSes.
* executor: make syscall table and number constantDmitry Vyukov2018-06-071-2/+2
| | | | | | | | | We see some crashes that suggest corruption of the syscall number: invalid command number 1296 (errno 11) invalid command number 107 (errno 110) Make the table and the number constant to prevent corruption.
* sys: move generate files to separate packagesDmitry Vyukov2018-05-051-8/+8
| | | | | | | | | Move generated files to gen subdir. This allows to: 1. Rebuild init.go without rebuilding generated code. 2. Excluding generated files from gometalinter checking. This makes faster and consume less memory. Update #538
generated by ggit (джигит) mrf-deployment (git 2.46.0) at 2026-03-11 19:49:44 +0000