0
Fork 0
mirror of https://codeberg.org/ziglang/zig synced 2026-11-18 07:05:05 +00:00
zig/lib/std/Build/Cache.zig

1580 lines
58 KiB

2023-12-11 23:08:03 +01:00
2020-03-06 20:23:15 -07:00
2020-05-01 23:06:10 -06:00
link.Elf: eliminate an O(N^2) algorithm in flush() Make shared_objects a StringArrayHashMap so that deduping does not need to happen in flush. That deduping code also was using an O(N^2) algorithm, which is not allowed in this codebase. There is another violation of this rule in resolveSymbols but this commit does not address it. This required reworking shared object parsing, breaking it into independent components so that we could access soname earlier. Shared object parsing had a few problems that I noticed and fixed in this commit: * Many instances of incorrect use of align(1). * `shnum * @sizeOf(elf.Elf64_Shdr)` can overflow based on user data. * `@divExact` can cause illegal behavior based on user data. * Strange versyms logic that wasn't present in mold nor lld. The logic was not commented and there is no git blame information in ziglang/zig nor kubkon/zld. I changed it to match mold and lld instead. * Use of ArrayList for slices of memory that are never resized. * finding DT_VERDEFNUM in a different loop than finding DT_SONAME. Ultimately I think we should follow mold's lead and ignore this integer, relying on null termination instead. * Doing logic based on VER_FLG_BASE rather than ignoring it like mold and LLD do. No comment explaining why the behavior is different. * Mutating the original ELF symbols rather than only storing the mangled name on the new Symbol struct. I noticed something that I didn't try to address in this commit: Symbol stores a lot of redundant information that is already present in the ELF symbols. I suspect that the codebase could benefit from reworking Symbol to not store redundant information. Additionally: * Add some type safety to std.elf. * Eliminate 1-3 file system reads for determining the kind of input files, by taking advantage of file name extension and handling error codes properly. * Move more error handling methods to link.Diags and make them infallible and thread-safe * Make the data dependencies obvious in the parameters of parseSharedObject. It's now clear that the first two steps (Header and Parsed) can be done during the main Compilation pipeline, rather than waiting for flush().
2024-10-11 23:28:31 -07:00
Improvements to docs and text * docs(std.math): elaborate on difference between absCast and absInt * docs(std.rand.Random.weightedIndex): elaborate on likelihood I think this makes it easier to understand. * langref: add small reminder * docs(std.fs.path.extension): brevity * docs(std.bit_set.StaticBitSet): mention the specific types * std.debug.TTY: explain what purpose this struct serves This should also make it clearer that this struct is not supposed to provide unrelated terminal manipulation functionality such as setting the cursor position or something because terminals are complicated and we should keep this struct simple and focused on debugging. * langref(package listing): brevity * langref: explain what exactly `threadlocal` causes to happen * std.array_list: link between swapRemove and orderedRemove Maybe this can serve as a TLDR and make it easier to decide. * PrefetchOptions.locality: clarify docs that this is a range This confused me previously and I thought I can only use either 0 or 3. * fix typos and more * std.builtin.CallingConvention: document some CCs * langref: explain possibly cryptic names I think it helps knowing what exactly these acronyms (@clz and @ctz) and abbreviations (@popCount) mean. * variadic function error: add missing preposition * std.fmt.format docs: nicely hyphenate * help menu: say what to optimize for I think this is slightly more specific than just calling it "optimizations". These are speed optimizations. I used the word "performance" here.
2023-04-23 20:06:21 +02:00
stage2: caching system integration & Module/Compilation splitting * update to the new cache hash API * std.Target defaultVersionRange moves to std.Target.Os.Tag * std.Target.Os gains getVersionRange which returns a tagged union * start the process of splitting Module into Compilation and "zig module". - The parts of Module having to do with only compiling zig code are extracted into ZigModule.zig. - Next step is to rename Module to Compilation. - After that rename ZigModule back to Module. * implement proper cache hash usage when compiling C objects, and properly manage the file lock of the build artifacts. * make versions optional to match recent changes to master branch. * proper cache hash integration for compiling zig code * proper cache hash integration for linking even when not compiling zig code. * ELF LLD linking integrates with the caching system. A comment from the source code: Here we want to determine whether we can save time by not invoking LLD when the output is unchanged. None of the linker options or the object files that are being linked are in the hash that namespaces the directory we are outputting to. Therefore, we must hash those now, and the resulting digest will form the "id" of the linking job we are about to perform. After a successful link, we store the id in the metadata of a symlink named "id.txt" in the artifact directory. So, now, we check if this symlink exists, and if it matches our digest. If so, we can skip linking. Otherwise, we proceed with invoking LLD. * implement disable_c_depfile option * add tracy to a few more functions
2020-09-13 19:17:58 -07:00
2020-04-07 23:57:19 -06:00
zig build system: change target, compilation, and module APIs Introduce the concept of "target query" and "resolved target". A target query is what the user specifies, with some things left to default. A resolved target has the default things discovered and populated. In the future, std.zig.CrossTarget will be rename to std.Target.Query. Introduces `std.Build.resolveTargetQuery` to get from one to the other. The concept of `main_mod_path` is gone, no longer supported. You have to put the root source file at the module root now. * remove deprecated API * update build.zig for the breaking API changes in this branch * move std.Build.Step.Compile.BuildId to std.zig.BuildId * add more options to std.Build.ExecutableOptions, std.Build.ObjectOptions, std.Build.SharedLibraryOptions, std.Build.StaticLibraryOptions, and std.Build.TestOptions. * remove `std.Build.constructCMacro`. There is no use for this API. * deprecate `std.Build.Step.Compile.defineCMacro`. Instead, `std.Build.Module.addCMacro` is provided. - remove `std.Build.Step.Compile.defineCMacroRaw`. * deprecate `std.Build.Step.Compile.linkFrameworkNeeded` - use `std.Build.Module.linkFramework` * deprecate `std.Build.Step.Compile.linkFrameworkWeak` - use `std.Build.Module.linkFramework` * move more logic into `std.Build.Module` * allow `target` and `optimize` to be `null` when creating a Module. Along with other fields, those unspecified options will be inherited from parent `Module` when inserted into an import table. * the `target` field of `addExecutable` is now required. pass `b.host` to get the host target.
2023-12-02 21:51:34 -07:00
2023-12-11 23:08:03 +01:00
WIP: move many global settings to become per-Module Much of the logic from Compilation.create() is extracted into Compilation.Config.resolve() which accepts many optional settings and produces concrete settings. This separate step is needed by API users of Compilation so that they can pass the resolved global settings to the Module creation function, which itself needs to resolve per-Module settings. Since the target and other things are no longer global settings, I did not want them stored in link.File (in the `options` field). That options field was already a kludge; those options should be resolved into concrete settings. This commit also starts to work on that, deleting link.Options, moving the fields into Compilation and ObjectFormat-specific structs instead. Some fields were ephemeral and should not have been stored at all, such as symbol_size_hint. The link.File object of Compilation is now a `?*link.File` and `null` when -fno-emit-bin is passed. It is now arena-allocated along with Compilation itself, avoiding some messy cleanup code that was there before. On the command line, it is now possible to configure the standard library itself by using `--mod std` just like any other module. This meant that the CLI needed to create the standard library module rather than having Compilation create it. There are a lot of changes in this commit and it's still not done. I didn't realize how quickly this changeset was going to balloon out of control, and there are still many lines that need to be changed before it even compiles successfully. * introduce std.Build.Cache.HashHelper.oneShot * add error_tracing to std.Build.Module * extract build.zig file generation into src/Builtin.zig * each CSourceFile and RcSourceFile now has a Module owner, which determines some of the C compiler flags.
2023-12-10 15:25:06 -07:00
2023-12-11 23:08:03 +01:00
2020-04-07 23:57:19 -06:00
2020-05-01 23:06:10 -06:00
2020-04-07 23:57:19 -06:00
2020-04-07 23:57:19 -06:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
2025-07-01 09:27:14 -07:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
std.Build.Cache: fix several bugs Aside from adding comments to document the logic in `Cache.Manifest.hit` better, this commit fixes two serious bugs. The first, spotted by Andrew, is that when upgrading from a shared to an exclusive lock on the manifest file, we do not seek it back to the start. This is a simple fix. The second is more subtle, and has to do with the computation of file digests. Broadly speaking, the goal of the main loop in `hit` is to iterate the files listed in the manifest file, and check if they've changed, based on stat and a file hash. While doing this, the `bin_digest` field of `std.Build.Cache.File`, which is initially `undefined`, is populated for all files, either straight from the manifest (if the stat matches) or recomputed from the file on-disk. This file digest is then used to update `man.hash.hasher`, which is building the final hash used as, for instance, the output directory name when the compiler emits into the cache directory. When `hit` returns a cache miss, it is expected that `man.hash.hasher` includes the digests of all "initial files"; that is, those which have been already added with e.g. `addFilePath`, but not those which will later be added with `addFilePost` (even though the manifest file has told us about some such files). Previously, `hit` was using the `unhit` function to do this in a few cases. However, this is incorrect, because `hit` assumes that all files already have their `bin_digest` field populated; this function is only valid to call *after* `hit` returns. Instead, we need to actually compute the hashes which haven't yet been populated. Even if this logic has been working, there was still a bug here, because we called `unhit` when upgrading from a shared to an exclusive lock, writing the (potentially `undefined`) file digests, but the loop itself writes the file digests *again*! All in all, the hashing logic here was actually incredibly broken. I've taken the opportunity to restructure this section of the code into what I think is a more readable format. A new function, `hitWithCurrentLock`, uses the open manifest file to try and find a cache hit. It returns a tagged union which, in the miss case, tells the caller (`hit`) how many files already have their hash populated. This avoids redundant work recomputing the same hash multiple times in situations where the lock needs upgrading. This also eliminates the outer loop from `hit`, which was a little confusing because it iterated no more than twice! The bugs fixed here could manifest in several different ways depending on how contended file locks were satisfied. Most notably, on a cache miss, the Zig compiler might have written the compilation output to the incorrect directory (because it incorrectly constructed a hash using `undefined` or repeated file digests), resulting in all future hits on this manifest causing `error.FileNotFound`. This is #23110. I have been able to reproduce #23110 on `master`, and have not been able to after this commit, so I am relatively sure this commit resolves that issue. Resolves: #23110
2025-04-24 01:17:32 +01:00
2024-03-23 03:58:32 +01:00
2024-03-23 03:58:32 +01:00
2020-04-11 16:01:17 -06:00
2024-03-23 03:58:32 +01:00
2020-05-01 23:06:10 -06:00
2024-03-23 03:58:32 +01:00
2024-03-23 03:58:32 +01:00
2024-03-23 03:58:32 +01:00
2024-03-23 03:58:32 +01:00
2020-09-19 14:16:58 +03:00
2020-09-19 14:16:58 +03:00
2021-11-24 23:08:37 -07:00
2020-04-07 23:57:19 -06:00
2022-04-18 23:08:00 -07:00
2022-04-18 23:08:00 -07:00
2020-05-01 23:06:10 -06:00
2020-03-05 22:59:19 -07:00
2020-04-11 16:01:17 -06:00
2020-03-05 22:59:19 -07:00
2020-03-05 22:59:19 -07:00
2020-03-05 22:59:19 -07:00
2023-12-11 23:08:03 +01:00
2020-03-05 22:59:19 -07:00
2020-03-05 22:59:19 -07:00
2020-03-05 22:59:19 -07:00
2020-03-05 22:59:19 -07:00
2020-04-11 16:01:17 -06:00
2020-05-01 23:06:10 -06:00
2023-12-11 23:08:03 +01:00
2020-05-01 23:06:10 -06:00
2023-12-11 23:08:03 +01:00
2023-12-11 23:08:03 +01:00