This fixes what was previously a guaranteed PathAlreadyExists error when LoadManifestOptions.dir (the build_root) was not equal to CWD, since the file would always get written to the CWD but then looked for in the build root, so after creation the retry would fail and then the re-creation would fail since it's created with an exclusive lock.
ensureUnusedCapacity should be preferred over ensureTotalCapacity.
By using ensureTotalCapacity the parsing could fail the assumption that there is enough unused capacity if node.string_value would already have elements.
However afaict both functions were only called with string_value = .empty. Therefore behavior should remain unchanged.
It cannot yet handle the behavior tests or compiler_rt tests due to them
using `@export` with `.internal`, but we can at least pass the standard
library and libc tests on x86_64.
The old test was relying on the following properties:
* There is a segment containing the ehdr, i.e. whose file offset is 0
* That segment is the first phdr with type `PT_LOAD`; because `PT_LOAD`
segments are sorted on `p_vaddr`, this means its virtual address is
the lowest of all loadable segments
The first assumption, although not standard, is somewhat reasonable, and
the commit prior to this one makes `Elf2` obey it. However, the second
assumption is not based on anything, and to my knowledge the linker is
entirely welcome to lay out the virtual address space as it sees fit.
Therefore, refactor this test to eliminate that bad assumption.
Although to my knowledge this is not strictly required by the format or
by any OS, it is highly conventional, and not doing so can definitely
break things in practice---including one of the standard library tests!
To achieve this, we add all of the segment nodes *before* the ehdr,
initializing rodata as a fixed "header" node in the `.elf` node, and
then the `.ehdr` node goes within that rodata segment node.
This is used on s390x and alpha. While `Elf2` doesn't yet support either
of those targets, we'll probably work on s390x soon, so it's worth
getting this support in now.
It turns out that SPARC has a strange definition of "PC-relative" where
the offsets are... unsigned? This results in some annoying restrictions
when linking: specifically, the GOT needs to appear after all code
segments, because these PC-relative relocations are used to access it.
When violating that, we should at least get relocation errors, but
because we were doing the wrong validation here, we were just triggering
miscompilations instead.
On SPARC, because JUMP_SLOT relocations write directly to the PLT, the
PLT is required to be RWX. Conventionally, this is achieved by placing
it in the "data" segment and making that RWX instead of RW, but that
just seems unnecessarily dangerous. Instead, let's try making a segment
specifically for the PLT.
Notably, this ensures that loadable segments are well-aligned in all
cases, and does so *without* requiring segments to have well-aligned
file offsets (which would result in needless bloat in the ELF file). It
also fixes a bunch of small alignment bugs across the backend, and adds
correct handling for the case of an input section / NAV / UAV forcing a
section, and possibly even segment, to increase its alignment.
Resolves: https://codeberg.org/ziglang/zig/issues/32135
This section, of type `SHT_HASH`, is a hash table used for looking up
symbol names in DSOs. It is referenced in `.dynamic` by the `DT_HASH`
entry. Without this, glibc assumes we export no symbols, and musl
refuses to load our DSOs at all!
Modern ELF systems broadly consider `SHT_HASH`/`DT_HASH` deprecated in
favour of `SHT_GNU_HASH`/`DT_GNU_HASH`, another hash table with a
different format which is more efficient for misses. However, as with so
many things in ELF, GNU's hash table format seems to be carefully
engineered to make incremental linking as difficult as possible. Every
dynamic linker out there continues to support `DT_HASH`, which actually
plays pretty nicely with incremental compilation, so I think we'll stick
with `DT_HASH` when doing incremental links.
I would now like to take a moment to complain about these hash tables.
These tables exist in order to make symbol lookups efficient when
applying relocations. The ldso implementations for both glibc and musl
libc work by iterating all relocations, and, for each one, looking up
the target symbol in each candidate DSO based on their `DT_HASH` or
`DT_GNU_HASH` table. The `DT_GNU_HASH` format came about to speed up
this lookup, since it's happening very often, so a small speedup can
lead to major improvements in load time.
However, I don't understand why nobody first solved the really obvious
inefficiency: every single symbol lookup is doing N different hash table
lookups where N is the number of loaded DSOs! A clearly better approach
for a dynamic linker to take is to have a single global hash table, and
every time a DSO is loaded, to add its symbols to that table. Then each
symbol lookup requires only *one* hash map lookup instead of N! Plus,
this change doesn't affect ELF files at all---it could be made in libc
implementations tomorrow. Yes, you need to iterate each DSO's dynamic
symbol table once in full, but that shouldn't be a big deal when you're
already iterating their *relocations* (those are far more numerous than
dynamic symbols!).
Here's another idea for free---after building the hash map, iterate
everyone's dynamic symbol table a second time, and build a lookup table
from their dynamic symbol index to the resolved symbol in the hash map.
Now relocation application doesn't even need a hash map; just a lookup
table access! Maybe this one isn't worthwhile, since the difference
between one LUT access and one hash map lookup isn't enough to warrant
the extra allocation and iteration, but it seems worth a try...
computers are fast, y'know!
What drives me a little crazy is that when someone ran into performance
problems with relocation application, instead of just spending an hour
speeding up lookups in ldso, they chose to invent lazy PLT binding. In
doing so, they opted to inflict needless complexity, unpredictable
runtime performance characteristics, unreportable failures, and security
risks in the form of a mutable jump table, upon every ELF system, for
ever. But I guess at least no poor soul had to implement a hash map.
But fine, let's look at status quo, and compare the standard `SHT_HASH`
format with `SHT_GNU_HASH`. As we all know, GNU's track record terms of
high-quality contributions to the ELF ecosystem is flawless, so there
must have been some good reasons for the changes.
Some small things which seem fine: they changed the hash function, which
I'll assume was with fair reason, and they added a bloom filter to allow
lookups to fail early (important if you accept the design of looking up
symbol names in individual DSOs).
The next change was to add a header field called `symoffset`, so that
the hash table can avoid wasting space on chains for the first N symbol
table entries which aren't actually global symbols. Unfortunately, it
seems that someone missed a memo, because this field, in a format which
debuted in 2006, has been redundant since the mid-90s. ELF files pretty
much universally use a separate `SHT_DYNSYM` section for the dynamic
symbol table, with the express purpose of *omitting* all the `STB_LOCAL`
symbols to save space. If a symbol doesn't appear in the hash table, it
shouldn't be in the dynamic symbol table to begin with. This quirk of
the `SHT_GNU_HASH` section isn't particularly *offensive*, it's just...
not useful.
Then the actual hash table representation. The "chains" array no longer
forms a linked list; instead, `chains[sym_idx]` now holds the *hash* for
the symbol's name. The linked list is eliminated in favour of an
assumption that symbols in the same bucket are contiguous in the symbol
table, and the least-significant bit of `chains[sym_idx]` is repurposed
to indicate the last symbol in a bucket (so you know when to stop
iterating).
In general I'm all for replacing linked lists with arrays, but this
particular change is really annoying. Setting aside the fact that it's
kinda odd for metadata *about* the symbol table to mandate a specific
ordering *within* the symbol table, this rigid ordering requirement is
also terrible for incremental linkers! It means that adding or removing
a symbol requires shifting potentially the entire symbol table up or
down to make sure the new symbol is in the right place, unless you're
happy having gaps all over the table (which I've been trying to avoid in
`Elf2`). I'm sure the contiguous-symbols assumption and the caching of
the hashes does improve lookup performance, but it seems like a far more
complicated solution than just doing some kind of caching in ldso
implementations. How this became the accepted solution to slow symbol
lookups is truly beyond me.
The old API fundamentally did not make sense, because the frontend does
not track exports incrementally: it only knows the set of referenced
exports at the end of each incremental update. Attempting to represent
this as an incremental add/remove-style API was just a lie. It was also
buggy; `deleteExport` was called from the main thread, leading to races
in incremental compilation if `deleteExport` was actually implemented!
We also exposed data in an unnecessarily complicated way which did not
benefit most linkers. The only one which actually cared about knowing
all exports for a given nav/uav upfront was the C linker (which here I
have written to reconstruct that data), and even it can probably be
refactored at some point to avoid that.
So, replace the export API with a simpler one, where all the frontend
does is tell the linker about every `Zcu.Export.Index` which is alive,
just before calling `flush`. In theory this information could actually
be passed into `flush`, but using a separate function made for a simpler
implementation in practice.
Resolves: https://github.com/ziglang/zig/issues/23616
`scalarize_bit_cast_vector_non_elementwise` was previously unconditionally
enabled on both big-endian and little-endian targets, which led to
suboptimal emitting of vector-to-integer bitcasts (https://zig.godbolt.org/z/qEG8zqfqn). This patch only
enables it on big-endian targets as the little-endian semantics match
the zig semantics.
Reviewed-on: https://codeberg.org/ziglang/zig/pulls/36512
Bug 1: `tryLock` can acquire a lock while another reader still holds
shared access to the lock. This one if fairly easy to reproduce, so the
test always catches it.
Bug 2: Cancelled writer can leave a stale semaphore permit, essentially
leaving the lock in an unusable state. This one is hard to reproduce
deterministically, the test catches it most of the time, but not always.
Let me know if you are OK with probabilistic tests like this, or I
should remove them.
Fixes https://codeberg.org/ziglang/zig/issues/36217
Fixes https://codeberg.org/ziglang/zig/issues/36178
The API for std.lang.Type changed in PR #35234, but the ASN.1 tag
encoder is using the old API. Update the check for an exhaustive enum
for the new API, and add a test that exercises this prong of the
encoder.