aboutsummaryrefslogtreecommitdiffstats
path: root/pkg/compiler/compiler.go
Commit message (Collapse)AuthorAgeFilesLines
* all: use any instead of interface{}Dmitry Vyukov2025-12-221-2/+2
| | | | Any is the preferred over interface{} now in Go.
* pkg/declextract: generated single openat for all related filesDmitry Vyukov2024-12-111-2/+2
|
* pkg/compiler: add automatic metaDmitry Vyukov2024-12-111-7/+2
| | | | | | Mark the whole file with "meta automatic" instead of marking each syscall. This reduces size of descriptions + allows to do special things with the whole file (e.g. we already treat auto consts specially).
* pkg/compiler: handle string syscall attributesFlorent Revest2024-12-091-10/+27
|
* pkg/compiler: allow recursion via arraysDmitry Vyukov2024-11-181-5/+7
| | | | | Permit structs to recursively contain itself in arrays. This is needed for netlink. Amusingly several netlink policies contain itself.
* pkg/compiler: add consts to all files that mention themDmitry Vyukov2024-11-131-0/+2
| | | | | | | | | We already do this in most cases except for template structs (nlattr notably). Add consts that are used in template structs to all files that use them. This helps to avoid flakiness, and allows to replace descriptions files with other descriptions files without regenerating all const files. This also fixes check for presence of descriptions for sys/linux/auto.txt.json.
* pkg/compiler: support if[expr] attributesAleksandr Nogikh2024-02-191-18/+55
| | | | | | | | | | | | | | | | | | | The expression may either include integers/consts or reference other fields in the structure via value[field1:field2:field3]. The fields on this path must all belong to structures and must not have any if conditions themselves. For unions, mandate that the last field has no conditions (it will be the default one). For structs, convert conditional fields into fields of a union type of the following form: anonymous_union [ value T (if[expression]) void void ]
* compiler: support nested string flags definitionsPaul Chaignon2023-12-051-0/+15
| | | | | | | | | This commit adds support for flags definitions such as: flags1 = "string1", "string2" flags2 = flags1, "string3" Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
* compiler: refactor recurFlattenFlagsPaul Chaignon2023-12-051-12/+18
| | | | | | | | | This commit refactors recurFlattenFlags using Go generics and new interfaces so that it also applies to a different set of flags types. In a subsequent commit, we will use that to perform the same recursive flattening for string flags. Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
* compiler: error on circular dependencies in flag definitionsPaul Chaignon2023-12-051-3/+10
| | | | | | | | To detect those circular dependencies, we simply keep track of which flags we already visited when flattening the flags definition. Suggested-by: Aleksandr Nogikh <nogikh@google.com> Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
* compiler: support nested int flags definitionsPaul Chaignon2023-12-051-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit adds support for flags definitions such as: flags1 = VAL1, VAL2 flags2 = flags1, VAL3 This is achieved by flattening nested flag definitions as part of the compilation. That is, nested flags are compiled into their fully unreferenced/unnested form. This flattening cannot be achieved in a single pass over the flags because we don't have a guarantee that we will encounter the subflags (ex. flags1 above) before their superflags (ex. flags2 above). Instead, in a first pass, we need to build an indexing of flags that we can use to flatten superflags in a second pass. Thankfully, this indexing is already computed in the form of comp.intFlags. Thus, we only need to implement the second pass, done with function compiler.flattenFlags(). This second pass walks the flag definitions recursively in an attemp to fully flatten them. It errors out if a flag definition has more than 5 levels of nested definitions. Being able to error in that way requires a bit of care in how we flatten the flags. Consider the following example where flags1 to flags5 have less than 5 leves of nesting, whereas flags6 should cause an error. flags6 = VAL6, flags5 flags5 = VAL5, flags6 ... flags1 = VAL1 If we were to flatten the flag definitions in place, then we might walk into flags5 first and fully flatten it. As a result, when we would walk into flags6, we would detect a single level of nesting and wouldn't error out. To avoid that, we work on the original set of nested flags and copy the flattened flags over only once we're done with all flags. Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
* compiler: prohibit homonymous flags and constsPaul Chaignon2023-11-281-1/+1
| | | | | | | | Since both flags and consts can be used as type-options for integers, we want to avoid ambiguity by preventing a flag and a const from having the same name. Signed-off-by: Paul Chaignon <paul.chaignon@gmail.com>
* sys: refactor const extractionAleksandr Nogikh2023-10-041-15/+5
| | | | | 1) Make FabricateSyscallConsts() operate on ConstFile. 2) Expose Pos inside ConstInfo.
* pkg/ast, pkg/compiler: support per-file metadataDmitry Vyukov2022-04-291-0/+20
| | | | | | | | | | | | | | | | | | | | | We have a bunch of hacks in syz-extract, syz-sysgen and syz-check with respect to description files unsupported on some arches, or that must not be part of make extract. Add 2 meta attribtues to files: meta noextract Tells `make extract` to not extract constants for this file. Though, `syz-extract` can still be invoked manually on this file. meta arches["arch1", "arch2"] Restricts this file only to the given set of architectures. `make extract` and ``make generate` will not use it on other architectures. Later we can potentially use meta attributes to specify git tree/commit that must be used for extraction. Maybe something else. Fixes #2754
* pkg/compiler: fix almost infinite recursion in template instantiationDmitry Vyukov2021-10-051-22/+24
| | | | | Fix another cases where recurion is finite but fan out factor is large, so our recursion check would take 5^10 iterations to handle this.
* pkg/compiler: simplify and enhance handling of builtinsDmitry Vyukov2020-05-051-7/+1
| | | | | | | | | Currently we have special support for each type of builtin node. This is complex and does not scale (we may want other types in future). Prepend the builtin descriptions to the user descriptions instead. This requires a bit of special support, like not reporting any builtin descriptions as unused, but otherwise much simpler and more flexible. Does not produce any diff in generated descriptions.
* prog: remove StructDescDmitry Vyukov2020-05-031-11/+7
| | | | | | | | | | | | Remove StructDesc, KeyedStruct, StructKey and all associated logic/complexity in prog and pkg/compiler. We can now handle recursion more generically with the Ref type, and Dir/FieldName are not a part of the type anymore. This makes StructType/UnionType simpler and more natural. Reduces size of sys/linux/gen/amd64.go from 5201321 to 4180861 (-20%). Update #1580
* prog: introduce Field typeDmitry Vyukov2020-05-021-3/+3
| | | | | | | | | | | | | Remvoe FieldName from Type and add a separate Field type that holds field name. Use Field for struct fields, union options and syscalls arguments, only these really have names. Reduces size of sys/linux/gen/amd64.go from 5665583 to 5201321 (-8.2%). Allows to not create new type for squashed any pointer. But main advantages will follow, e.g. removing StructDesc, using TypeRef in Arg, etc. Update #1580
* prog: remove Dir from TypeDmitry Vyukov2020-05-011-4/+3
| | | | | | | | | | | | | | | | | | Having Dir is Type is handy, but forces us to duplicate lots of types. E.g. if a struct is referenced as both in and out, then we need to have 2 copies and 2 copies of structs/types it includes. If also prevents us from having the struct type as struct identity (because we can have up to 3 of them). Revert to the old way we used to do it: propagate Dir as we walk syscall arguments. This moves lots of dir passing from pkg/compiler to prog package. Now Arg contains the dir, so once we build the tree, we can use dirs as before. Reduces size of sys/linux/gen/amd64.go from 6058336 to 5661150 (-6.6%). Update #1580
* pkg/compiler: deduplicate Types in descriptionsDmitry Vyukov2020-04-261-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add prog.Ref Type that serves as a proxy for real types and allows to deduplicate Types in generated descriptions. The Ref type is effectively an index in an array of types. Just before serialization pkg/compiler replaces real types with the Ref types and prepares corresponding array of real types. When a Target is registered in prog package, we do the opposite operation and replace Ref's with the corresponding real types. This brings improvements across the board: compiler memory consumption is reduced by 15%, test building time by 25%, descriptions size by 33%. Before: $ du -h sys/linux/gen 54M sys/linux/gen $ time GOMAXPROCS=1 go test -p=1 -c ./prog real 0m54.200s real 0m53.883s $ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog real 0m27.911s real 0m27.767s $ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen 20.59 100% 3200016 20.97 100% 3445976 20.25 100% 3209684 After: $ du -h sys/linux/gen 36M sys/linux/gen $ time GOMAXPROCS=1 go test -p=1 -c ./prog real 0m42.290s real 0m43.230s $ time GOMAXPROCS=1 go install -p=1 ./tools/syz-execprog real 0m24.337s real 0m24.727s $ TIME="%e %P %M" GOMAXPROCS=1 time go tool compile ./sys/linux/gen 19.11 100% 2764952 19.66 100% 2787624 19.35 100% 2749376 Update #1580
* all: fix liner errorsDmitry Vyukov2020-04-191-8/+8
| | | | | | | | | | | | pkg/compiler/compiler.go:182: line is 125 characters func (comp *compiler) parseAttrs(descs map[string]*attrDesc, parent ast.Node, attrs []*ast.Type) (res map[*attrDesc]uint64) { sys/targets/common.go:47:21: unnecessary conversion makeMmap(^uint64(target.PageSize)+1, target.PageSize, 0), ^ sys/targets/common.go:61: File is not `gofmt`-ed with `-s` &prog.Call{ sys/windows/init.go:35: File is not `gofmt`-ed with `-s` &prog.Call{
* pkg/compiler: error on duplicate attributesDmitry Vyukov2020-04-191-0/+4
|
* pkg/compiler: refactor attribute handlingDmitry Vyukov2020-04-191-61/+37
| | | | | | | | | | | | Introduce common infrastructure for describing and parsing attribute instead of custom per-attribute code scattered across several locations. Change align attribute syntax from the weird align_N to align[N]. This also allows to use literal constants as N. Introduce notion of builtin constants. Currently we have only PTR_SIZE, which is needed to replace align_ptr with align[PTR_SIZE].
* pkg/compiler: emit warnings after generate stageDmitry Vyukov2020-03-171-3/+3
| | | | | | Description generation can also produce errors. We don't want to emit warnings if there are any errors. Move warnings emission to the very end of compilation.
* pkg/compiler: support complex len targetsDmitry Vyukov2019-05-141-0/+10
| | | | | | | | | | This change adds compiler support for complex path expressions in len targets. E.g. it allows to refer to a sibling field as len[parent_struct:field:another_field]. See the docs change for details. This is just a compiler change. The feature is not yet supported by the prog package.
* pkg/ast: refactor COLON handlingDmitry Vyukov2019-05-141-1/+1
| | | | | | | This prepared for handling of bytesize[parent:foo:bar] expressions by allowing multiple identifiers after colon. No functional changes for now, just preparation for storing more than one identifier after colon.
* pkg/compiler: add error handler in CollectUnusedMarco Vanotti2018-11-211-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * pkg/compiler: Add error handler in `CollectUnused`. This commit adds an error handler for the `CollectUnused` function. The error handler just panics on any error, but is useful for debugging. The error handler is used any time `comp` finds an error, and if it's missing, it will panic due to a `nil` pointer dereference. At least now we get a better understanding of the errors. The only user of `CollectUnused` is `sys/fuchsia/fidlgen`, which is failing now and will be fixed in a future commit. The output message looks like this: ``` panic: could not collect unused nodes. fidl_net-stack.txt:110:15: unknown type zx_chan_zircon_ethernet_Device_client ``` * pkg/compiler Better error handling in CollectUnused This commit changes the default error handler for compiler to `ast.LoggingHandler`, meaning that if `nil` is passed as an error handler, `LoggingHandler` will be used instead. `CollectUnused` now returns an error if any of the subfunctions produce errors. `fidlgen` is the only caller of `CollectUnused`, and now checks for errors as well. * pkg/compiler Add tests for CollectUnused This commit adds basic tests for the CollectUnused function. There's one test that checks that it returns the right nodes, and another one that makes sure that it returns errors when needed. To make the test clearer, I had to add the error handler as an explicit parameter in `CollectUnunsed`, instead of using the default one. This avoid printing garbage in the logs. The `TestCollectUnusedError` function uses a nopErrorHandler to avoid printing anything. * pkg/compiler fix presubmit warnings
* sys/fuchsia: prune unused structs in syscall description generated by fidlgenDokyung Song2018-09-111-6/+11
| | | | | | | | After generating syscall description for fidl files using fidlgen, prune all unused structs using the exact same mechanism used by the compiler's check for unused structs. This allows the FIDL compiler to support modular compilation; it does not need to have global knowledge of whether each struct is used or not.
* prog, pkg/compiler: support fmt typeDmitry Vyukov2018-07-081-1/+1
| | | | | fmt type allows to convert intergers and resources to string representation.
* pkg/compiler: check for unused declarationsDmitry Vyukov2018-06-301-7/+10
| | | | | Error on unused structs/unions/resources/flags. Finds tons of bugs.
* gometalinter: some fixes for unparamDmitry Vyukov2018-05-031-3/+3
| | | | | | But we still can't enable it as there are more [uninteresting] warnings. Update #538
* pkg/compiler: support non-zero terminated filenamesDmitry Vyukov2018-04-021-2/+5
| | | | | | | | | | | | | | | Now file names become: string[filename] with a possibility of using other string features: stringnoz[filename] string[filename, CONST_SIZE] and filename is left as type alias as it is commonly used: type filename string[filename]
* all: fix gometalinter warningsDmitry Vyukov2018-03-081-1/+1
| | | | Fix typos, non-canonical code, remove dead code, etc.
* pkg/compiler: support size attribute for unionsDmitry Vyukov2018-03-051-20/+30
|
* pkg/compiler: add size attribute for structsDmitry Vyukov2018-03-051-6/+52
| | | | The size attribute allows to pad a struct up to the specified size.
* pkg/compiler: switch attributes from Ident to TypeDmitry Vyukov2018-03-051-8/+14
| | | | | This allows parametrized attributes like size[10]. But this is not used for now.
* sys/syz-sysgen: don't generate syz_ syscall numbersDmitry Vyukov2018-01-131-1/+3
| | | | They don't seem to be used today.
* pkg/compiler: support type templatesDmitry Vyukov2018-01-131-6/+60
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Netlink descriptions contain tons of code duplication, and need much more for proper descriptions. Introduce type templates to simplify writing such descriptions and remove code duplication. Note: type templates are experimental, have poor error handling and are subject to change. Type templates can be declared as follows: ``` type buffer[DIR] ptr[DIR, array[int8]] type fileoff[BASE] BASE type nlattr[TYPE, PAYLOAD] { nla_len len[parent, int16] nla_type const[TYPE, int16] payload PAYLOAD } [align_4] ``` and later used as follows: ``` syscall(a buffer[in], b fileoff[int64], c ptr[in, nlattr[FOO, int32]]) ```
* pkg/ast: refactor WalkDmitry Vyukov2018-01-111-1/+1
| | | | | Refactor Walk so that it's possible to abort or wrap walk of child nodes. Will be needed for future changes.
* pkg/compiler: add builtin bool type aliasesDmitry Vyukov2018-01-081-0/+3
| | | | | | | | | | | | | | | | This adds builtin: type bool8 int8[0:1] type bool16 int16[0:1] type bool32 int32[0:1] type bool64 int64[0:1] type boolptr intptr[0:1] We used to use just int's for bools. But bool types provide several advantages: - make true/false probability equal - improve description expressiveness - reduce search space (we will take advantage of this later)
* sys: support type aliases (aka typedefs)Dmitry Vyukov2018-01-081-0/+5
| | | | | | | | | | | | | | | | | | | | | | Complex types that are often repeated can be given short type aliases using the following syntax: ``` type identifier underlying_type ``` For example: ``` type signalno int32[0:65] type net_port proc[20000, 4, int16be] ``` Then, type alias can be used instead of the underlying type in any contexts. Underlying type needs to be described as if it's a struct field, that is, with the base type if it's required. However, type alias can be used as syscall arguments as well. Underlying types are currently restricted to integer types, `ptr`, `ptr64`, `const`, `flags` and `proc` types.
* all: more assorted fuchsia supportDmitry Vyukov2017-09-221-2/+5
|
* pkg/compiler: don't genererate missing syscallsDmitry Vyukov2017-09-151-0/+2
| | | | | | | | We used to generate them only because manager had no idea what arch it is testing. So syscalls numbers had to match between all arches. This is not needed anymore. Also don't generate unreferenced structs/resources.
* prog, sys: move types to progDmitry Vyukov2017-09-051-13/+13
| | | | | | | | | | | Large overhaul moves syscalls and arg types from sys to prog. Sys package now depends on prog and contains only generated descriptions of syscalls. Introduce prog.Target type that encapsulates all targer properties, like syscall list, ptr/page size, etc. Also moves OS-dependent pieces like mmap call generation from prog to sys. Update #191
* sys: rename Call to SyscallDmitry Vyukov2017-09-051-1/+1
| | | | | In preparation for moving sys types to prog to avoid confusion between sys.Call and prog.Call.
* sys, pkg/compiler: move padding computation to compilerDmitry Vyukov2017-09-041-26/+33
| | | | This makes types constant during execution, everything is precomputed.
* pkg/compiler: use correct arch ptr sizeDmitry Vyukov2017-09-041-2/+2
|
* pkg/compiler: move checking code to a separate fileDmitry Vyukov2017-09-041-390/+0
|
* pkg/compiler: detect recursive struct declarationsDmitry Vyukov2017-09-041-22/+95
| | | | Update #217
* pkg/compiler: check and generate typesDmitry Vyukov2017-09-021-258/+323
| | | | | | Move most of the logic from sysgen to pkg/compiler. Update #217
generated by ggit (джигит) mrf-deployment (git 2.46.0) at 2026-03-12 00:29:25 +0000