This commit implements the `@backingInt` and `@fromBackingInt` builtins.
These builtins are meant to replace the now-deprecated `@intFromEnum` and
`@enumFromInt` builtins.
`@backingInt` works with all enums and with bitpacks with *explicit* backing
integer types only. It also works with tagged unions, returning the backing
integer of the active tag value.
An `undefined` enum or bitpack yields an `undefined` backing integer.
The builtin is lowered to `bit_cast`.
`@fromBackingInt` infers its result type, which may be any enum or a bitpack
with an *explicit* backing integer type. It takes a parameter of exactly
that backing integer type.
For enums, passing a backing integer that is either `undefined` or would
yield an invalid tag value results in safety-checked Illegal Behavior.
For bitpacks, passing an `undefined` backing integer yields an `undefined`
bitpack.
The builtin is lowered to `bit_cast` or `bit_cast_safe` (more details below).
`@bitCast` now also performs a safety check for invalid tag values if its
destination type is an `enum`.
This commit also introduces a new AIR instruction, `bit_cast_safe`.
Similarly to `int_cast_safe`, it includes a safety check for invalid enum
tag values. When `@enumFromInt` is eventually removed, `int_cast_safe` won't
need to perform this safety check anymore.
Legalize only implements `expand_bit_cast_safe`, and no additional insts
to scalarize `bit_cast_safe` with vector/array operands. That is because
it can just reuse the existing scalarizations for regular `bit_cast`.
The safety check mandated by `bit_cast_safe` is only desired if the result
type is a scalar enum tag value, so the scalarization process is exactly
equivalent. The only difference is that we have to perform a safety check
on the result of the scalarization.
This means that there's no reason for a backend to want scalarization of
regular `bit_cast`, but not of `bit_cast_safe` (or the other way round).
If a backend is having trouble with the `bit_cast_safe` safety check it
should just use `expand_bit_cast_safe` instead.
Also adds a `std.meta.BackingInt()` function to conveniently get the result
type of `@backingInt` and for use in the langref.
The SPIR-V backend previously ran codegen single-threaded in the linker thread.
Now each codegen job creates an `Mir` like other self-hosted backends.
Other changes:
- Bring back `dedup_types.zig` and `prune_unused.zig` **ISel**s which were originaly
removed because codegen was single-threaded at that time and therefor had no use
- Clean up `BinaryModule.zig`
- Remove `checkLogicalPtrOperation` from `elemPtrOneLayerOnly` in `Sema.zig`.
Element access uses `OpAccessChain`, which works on all logical address spaces
without `VariablePointers`. the check is only needed for pointer arithmetic.
Closes#35240Fixes#35238Fixes#35259
Supported types are:
- `OpTypeSampler`
- `OpTypeImage`
- `OpTypeSampledImage`
- `OpTypeRuntimeArray` with indexing and `.len` field
The SPIR-V backend is bit-rotted so behavior tests no longer pass (compiler crashes).
However I've verified the new added tests are passing.
All public codegen and linker APIs now use the following error set:
Allocator.Error || Io.Cancelable || error{AlreadyReported}
This is defined as `link.Error` and aliased as `codegen.Error`.
The compiler "backend" (including both codegen and linker) has a fairly
limited set of failure modes. Most of them are as follows:
* Bad inline assembly (codegen)
* Output file I/O error (link)
* Symbol with no definition or multiple definitions (link)
* Relocation error, e.g. overflow (link)
* Unimplemented/unsupported feature (codegen/link)
* `error.OutOfMemory` (codegen/link)
* `error.Canceled` (codegen/link)
The last two cases are special, because they are possible across most of
the compiler codebase and have fixed code paths for handling---e.g.
`error.OutOfMemory` almost always calls `Compilation.setAllocFailure`,
and `error.Canceled` should typically propagate all the way up the call
stack.
However, all of the other cases should follow the same general pattern:
the codegen/linker implementation should mark an error on `Compilation`
somehow (either through `link_diags` or `failed_codegen`), and return
`error.AlreadyReported`. This error code indicates that an operation
failed, but that the caller does not need to take action to recover,
because the failure has already been recorded in a way which will be
expressed to the user. For instance, the codegen error set used to
contain `error.Overflow`, but this case should have already been
reported to the user, because implementation-agnostic code cannot know
how best to express the failure to the user.
The `error.AlreadyReported` error code encompasses what was previously
represented by multiple errors, including `error.CodegenFail`,
`error.LinkFailure`, and `error.AnalysisFail`. It is not necessary to
differentiate these cases. (Not to be confused with the usage of
`error.AnalysisFail` in the compiler *frontend*, i.e. `Sema`, which is
unchanged by this diff.)
Because the backend error set is now very small, it is easy to
exhaustively `switch` on, and also many functions can be given concrete
error sets. There are no longer different error sets for different
operations. (As an exception, I have not tackled `link.File.open` in
this diff, which has a big inferred error set where I think most errors
are reported to the user with an `else => |e|` case.)
I have also changed how `link.MappedFile`, our abstraction for handling
the awkward "moving things around" aspect of incremental linking, does
error handling. The public API of this abstraction now uses the
following small error set:
Allocator.Error || Io.Cancelable || error{MappedFileIo}
The first two cases are self-explanatory. Then, `error.MappedFileIo` is
a catch-all error code encompassing that an error was encountered when
accessing the underlying `Io.File`. This error may have occurred when
attempting to flush the file to disk, or to change its size, etc. In
this case, the *specific* error---which was actually returned from the
`Io.File` API---is available in the `MappedFile.io_err.?` field. Linker
implementations which use `MappedFile` (currently `Elf2` and `Coff`)
should eventually handle `error.MappedFileIo` by reporting an error in
`Compilation.link_diags`, including that specific I/O error in the error
message.
The rationale here is essentially that error codes returned from
functions exist for control flow purposes, and a linker implementation
should not care about the distinction between file I/O error codes (e.g.
`error.DiskQuota` vs `error.InputOutput`). The only reason it needs the
concrete error is to expose it to the user. Therefore, by wrapping these
failure modes under `error.MappedFileIo`, we keep the error set actually
used by the linker implementation as small as possible, which encourages
avoiding patterns like `else => |e| diags.fail(...)` (which is
potentially dangerous since it may prevent `error.OutOfMemory` or
`error.Canceled` from propagating correctly). It additionally helps
linker implementations maintain more precise error messages---for
instance, if `Elf2` encounters `error.NoSpaceLeft` writing to the output
file, the error will now read "failed to write output file: NoSpaceLeft"
instead of something more generic like "prelink failed: NoSpaceLeft".
Finally, while working on these refactors, I was able to eliminate those
pesky `src_loc: Zcu.LazySrcLoc` parameters from the codegen and link
logic. These parameters did not make sense, because at this point in the
compiler pipeline, we do not have precise source location
information---so these parameters were always passed as just the
location of the function declaration, or as some placeholder
(`.unneeded` or the top of `lib/std/std.zig`) if there wasn't an
appropriate function definition. The only logic which actually *uses*
these source locations is codegen implementations' `fail` functions. Per
my earlier list of failure modes, those functions are called in two main
cases:
* Bad inline assembly
* Unimplemented/unsupported feature
The first case should be moved to the compiler frontend (tracked by
https://github.com/ziglang/zig/issues/10761), while the second case is
essentially a compiler TODO so it is permissible for it to have
imprecise error reporting. Therefore, codegen implementations' `fail`
functions now just call `navSrcLoc` themselves when necessary, which
(once we make improvements to inline assembly) will only happen when
there is a deficiency in the codegen implementation.
I was careful to make `Elf2` and `Coff` error reporting work well in
this diff, by using no `else` case other than `else => |e| return e`
when `switch`ing on errors and by always handling `error.MappedFileIo`
by unwrapping `MappedFile.io_err.?`. I also added concrete error sets to
almost every function in `Elf2`. (That was, um, actually why I started
this diff, because it was annoying me that I wouldn't get as many
compile errors at once...)
Sorry for the mega-commit, this diff got a little out of control.
The main thing here is a complete rework of how Elf2 handles the symbol
table. I messed around with the design for a while and landed on
something which is fairly memory-efficient (in particular the overhead
for STB_LOCAL symbols is as low as possible) and fulfils some of the
more awkward constraints of the ELF format. The main such constraint is
that all STB_LOCAL symbols in a symbol table are required to appear
before any STB_GLOBAL/STB_WEAK symbols. This is further complicated by
the fact that when producing a DSO, symbols with STV_HIDDEN or
STV_INTERNAL visibility are required to have STB_LOCAL binding in the
symbol table, even though they are global symbols from the perspective
of the link editor. Plus, when combining multiple symbols with the same
name, the resulting visibility is the strictest of all of the inputs, so
it is possible at any point in compilation to discover an extern/export
symbol which forces an existing STB_GLOBAL symbol to become STB_LOCAL
and therefore requires it to move to an earlier symtab index. Dealing
with all of this was quite awkward.
But I got there! I also implemented a lot of features in the process. I
don't remember everything perfectly, but here's a vague list:
* Multiple definitions of and/or unresolved references to symbols are
now combined correctly in all cases
* `.bss` sections from inputs are correctly lowered (we don't actually
emit a `.bss` section of our own yet, but I was able to put that data
into the `.data` section so that the functionality is correct)
* Relocations in link inputs are now always processed (previously they
would be silently ignored in most cases)
* Linker errors are triggered if a supported input section has a
relocation which targets an unsupported input section (previously
the unsupported section's symbol was dropped and associated
relocations would be silently ignored)
* When linking a static executable, an error is emitted if a required
symbol (i.e. an undefined reference with strong linkage) was never
defined
* Duplicate symbol errors now work correctly
* When emitting a relocatable, the offsets of relocation entries are now
correct (previously the offsets written were relative to a symbol
rather than a section, meaning that e.g. almost all text relocations
were just in a single function)
The changes in all of the other linkers and codegen backends are some
added type-safety at the codegen-linker API boundary. There are now
distinct `u32`-backed types for identifying an "atom" (the thing we're
codegenning) and a "symbol" (the thing which a relocation targets).
Linker implementations can use a couple of private helper functions to
convert between this implementation-agnostic type and their specific
type; for instance, `Elf2` can convert between a `Symbol.Id` and a
`link.File.SymbolId` with `Symbol.Id.fromTypeErased` and
`Symbol.Id.toTypeErased`. I didn't implement this nicely for any other
linker, so right now there's a lot of `@enumFromInt`/`@intFromEnum`
sprinkled all over the place, particularly with the legacy ELF and
Mach-O linkers.
I tested that I could still perform incremental updates to the Zig
compiler using this commit. In terms of the new behaviors, the most
interesting stuff is symbol and relocation resolution, so I ran a few
tests involving building a "Hello World" binary in various different
ways:
* `build-exe` correctly succeeds
* `build-exe -fno-compiler-rt` correctly reports undefined symbols
* `build-obj` linked with `build-exe` correctly succeeds
* `build-obj` linked with `build-exe -fno-compiler-rt` correctly reports
undefined symbols
* `build-obj -fcompiler-rt` linked with `build-exe -fno-compiler-rt`
correctly succeeds
* `build-obj -fcompiler-rt` linked with `build-exe` correctly succeeds
(the compiler-rt symbols are weak so the global symbols are
arbitrarily resolved to one of the two implementations)
I also manually verified with `readelf` that symbol tables were always
ordered correctly (before this PR, `readelf -s` would usually emit
warnings about incorrectly-ordered symtabs!), and verified that various
visibility attributes worked as expected.
No actual test coverage is added due to the current lack of a useful
linker test harness. Once a good test harness is available I will be
willing to write some tests.
I've realised that the cause of at least some of our weird CI flakiness
was a bug in how `Nav` values were resolved. Consider this scenario: the
frontend resolves the type of a `Nav`, and then sends a function to the
backend, which requires the backend to lower a pointer to that `Nav`.
The backend calls `InternPool.getNav` to determine the `Nav`'s type.
However, this races with the frontend resolving the *value* of that
`Nav`. This involves writing separately to two fields, `bits` and
`type_or_value`. If only one of these changes is observed, then the
backend will incorrectly interpret the type as the value or vice versa,
leading to a crash or even a miscompilation. (Of course, there's also
the straightforward issue that the racing loads were non-atomic, making
them illegal).
The only good solution to this was to make `Nav` 4 bytes bigger, giving
it separate `type` and `value` fields. In theory that's a quite small
change, but it ended up having a bunch of nice consequences which led to
this diff being a bit bulkier than expected:
* `Nav.Repr.Bits` was simplified, because it no longer has to track
"resolution status": we can use `.none` for that. This frees up some
bits to make things more consistent between the "type resolved" and
"fully resolved" states.
* This consistency allowed the `Nav.status` union to be replaced with a
simpler field `Nav.resolved`, which is a bit nicer to work with.
* Most of the "getter" functions were able to be removed from `Nav`
because the state they were fetching had been moved to simple fields
on `Nav.resolved`.
* There were still a handful of free bits in `Nav.Repr.Bits`, which
could be used to represent the "const" and "threadlocal" flags rather
than these being stored on `Key.Extern` and `Key.Variable`. This is a
bit more convenient for linkers.
* With those bits gone, `Key.Variable` is a trivial wrapper around a
type and an initial value, and the fact that a declaration is mutable
can be represented solely through the "const" flag. Therefore,
`Key.Variable` no longer served a purpose, and could be eliminated
entirely in favour of storing the variable's initial value directly in
the "value" field of the `Nav`.
So, I'm quite pleased with this refactor! But anyway, regarding the bug
fix which actually motivated this: if I've done my job correctly, this
should solve some crashes, such as these (which were what tipped me off
to this bug in the first place):
https://codeberg.org/ziglang/zig/actions/runs/2306/jobs/7/attempt/1https://codeberg.org/ziglang/zig/actions/runs/2173/jobs/6/attempt/1
...and, who knows, perhaps even the random SIGSEGVs we've seen on some
targets! Probably not, but one can hope.
Introduces a small abstraction, `link.DebugConstPool`, to deal with
lowering type/value information into debug info when it may not be known
until type resolution (which in some cases will *never* happen). It is
currently only used by self-hosted DWARF logic, but it will also be of
use to the LLVM backend (which is my next focus).
Now that https://github.com/ziglang/zig/issues/24657 has been
implemented, the compiler can simplify its internal representation of
comptime-known `packed struct` and `packed union` values. Instead of
storing them field-wise, we can simply store their backing integer
value. This simplifies many operations and improves efficiency in some
cases.
This commit expands on the foundations laid by https://github.com/ziglang/zig/pull/23177
and moves even more `Sema`-only functionality from `Value`
to `Sema.arith`. Specifically all shift and bitwise operations,
`@truncate`, `@bitReverse` and `@byteSwap` have been moved and
adapted to the new rules around `undefined`.
Especially the comptime shift operations have been basically
rewritten, fixing many open issues in the process.
New rules applied to operators:
* `<<`, `@shlExact`, `@shlWithOverflow`, `>>`, `@shrExact`: compile error if any operand is undef
* `<<|`, `~`, `^`, `@truncate`, `@bitReverse`, `@byteSwap`: return undef if any operand is undef
* `&`, `|`: Return undef if both operands are undef, turn undef into actual `0xAA` bytes otherwise
Additionally this commit canonicalizes the representation of
aggregates with all-undefined members in the `InternPool` by
disallowing them and enforcing the usage of a single typed
`undef` value instead. This reduces the amount of edge cases
and fixes a bunch of bugs related to partially undefined vecs.
List of operations directly affected by this patch:
* `<<`, `<<|`, `@shlExact`, `@shlWithOverflow`
* `>>`, `@shrExact`
* `&`, `|`, `~`, `^` and their atomic rmw + reduce pendants
* `@truncate`, `@bitReverse`, `@byteSwap`
This experimental target was never fully completed. The operating system
is not that interesting or popular anyway, and the maintainer is no
longer around.
Not worth the maintenance burden. This code can be resurrected later if
it is worth it. In such case it will be subject to greater scrutiny.
My original goal here was just to get the self-hosted Wasm backend
compiling again after the pipeline change, but it turned out that from
there it was pretty simple to entirely eliminate the shared state
between `codegen.wasm` and `link.Wasm`. As such, this commit not only
fixes the backend, but makes it the second backend (after CBE) to
support the new 1:N:1 threading model.
As of this commit, every backend other than self-hosted Wasm and
self-hosted SPIR-V compiles and (at least somewhat) functions again.
Those two backends are currently disabled with panics.
Note that `Zcu.Feature.separate_thread` is *not* enabled for the fixed
backends. Avoiding linker references from codegen is a non-trivial task,
and can be done after this branch.
The idea here is that instead of the linker calling into codegen,
instead codegen should run before we touch the linker, and after MIR is
produced, it is sent to the linker. Aside from simplifying the call
graph (by preventing N linkers from each calling into M codegen
backends!), this has the huge benefit that it is possible to
parallellize codegen separately from linking. The threading model can
look like this:
* 1 semantic analysis thread, which generates AIR
* N codegen threads, which process AIR into MIR
* 1 linker thread, which emits MIR to the binary
The codegen threads are also responsible for `Air.Legalize` and
`Air.Liveness`; it's more efficient to do this work here instead of
blocking the main thread for this trivially parallel task.
I have repurposed the `Zcu.Feature.separate_thread` backend feature to
indicate support for this 1:N:1 threading pattern. This commit makes the
C backend support this feature, since it was relatively easy to divorce
from `link.C`: it just required eliminating some shared buffers. Other
backends don't currently support this feature. In fact, they don't even
compile -- the next few commits will fix them back up.