diff --git a/docs/startup-snapshot.md b/docs/startup-snapshot.md index e9eff20a6..856923d72 100644 --- a/docs/startup-snapshot.md +++ b/docs/startup-snapshot.md @@ -37,8 +37,8 @@ all of these hold: top-level code has to run there for DevTools to see its console output and stop at its breakpoints; - `supportsStartupSnapshot()` accepts the Worker: ES-modules syntax (not a service-worker - script), no Wasm modules, not a Python Worker, the legacy module registry (no - `new_module_registry`), the `typescript_implemented_streams` flag, and only bindings of kinds + script), no Wasm modules, not a Python Worker, the `typescript_implemented_streams` flag, and + only bindings of kinds that transfer (`WorkerdApi::isBindingTransferableThroughSnapshot()`; see step 3 below). The C++ streams implementation leaves a pending `pull()` as a C++ promise continuation reachable from the context, which `CreateBlob()` cannot serialize; the TypeScript implementation keeps it in @@ -105,7 +105,10 @@ all of these hold: 6. The zygote's global scope is cleared, the script's module context handle dropped, and `IsolateBase::createSnapshotBlob()` runs. Its steps are numbered in `setup.c++`: 1. `bakeModuleRegistryIntoSnapshot()`: every materialized module handle of the legacy registry - is pinned and recorded in `artifact.moduleRecords`; the handles are reset. + is pinned and recorded in `artifact.moduleRecords`; the handles are reset. With the new + module registry, `modules::recordIsolateModuleRegistryForSnapshot()` instead pins the + `v8::Module` of each of the registry's resolutions (a context type and a specifier, query + and fragment included) and records them in `artifact.newModuleRegistryRecords`. 2. `WrappablesSerializeMetadata::recordLiveWrappables()`: every live JSG object is recorded while it still exists, by its tag alone for a stateless type, as a `snapshotClone()` otherwise (`artifact.liveWrappables`), keyed by its CppgcShim address. The metadata roots @@ -173,6 +176,12 @@ long as the isolate does. In order: (`tryTakeSnapshotModule()`) and still registers the lazily compilable builtins the zygote never touched, then `applySnapshotSeeds()`. Runtime `import()`, `require()` and the `Buffer`/`process` globals therefore resolve to the instances the top level evaluated against. + With the new module registry, `ModuleRegistry::attachToIsolate()` instead rebuilds the + isolate's resolution and instantiation tables from `newModuleRegistryRecords` + (`IsolateModuleRegistry::restoreFromSnapshot()`), so that `import()` and `import.meta` find + the modules the top level evaluated. Every synthetic module of the new registry is created + with the same evaluation steps, which `Worker::Isolate` registers as an external reference + (`getSyntheticModuleEvaluationStepsRef()`). 5. The `Worker` constructor takes `env` from the snapshot (`extractOnceFromSnapshot()`), complete with its bindings: `factory_new_worker()` compiles no bindings for a Worker started from a snapshot, while `ctx.exports` is still built for it afterwards, by `worker_set_ctx_exports()`. @@ -193,6 +202,7 @@ long as the isolate does. In order: | `Error.stackTraceLimit` as the top level left it, deleted fail-closed properties | pinned record, replayed after restore | `restoreTimeGlobalsIndex` | | `process`/`Buffer` values of the global scope | one pinned object | `globalScopeStateIndex` | | Materialized module-registry entries | pinned handles, seeded into the new registry | `moduleRecords` | +| New module registry's resolutions | pinned `v8::Module`s, rebuilt into the restored registry | `newModuleRegistryRecords` | | Global-scope event listeners | pinned identity, callback and receiver; re-registered in C++ | `listenerRecords` | Not transferred, by design: `ctx.exports` (built for the real Worker after the zygote has run), @@ -226,9 +236,10 @@ also overrides `Wrappable::snapshotReferences()` and `snapshotRelink()`, as `Cry `subtle`: the restored instance is built afresh, and would otherwise refer to a new object rather than to the one behind the restored wrapper. -A `jsg::Function` the top level keeps (the function `createRequire()` returns, for example) -transfers if its functor provides `Impl clone() const`; V8 patch 0052 keeps functions created by -`v8::Function::New()` in the context snapshot so that their wrapper goes through the same path. +A `jsg::Function` the top level keeps (the function `createRequire()` returns, or the new module +registry's `import.meta.resolve`, for example) transfers if its functor provides +`Impl clone() const`; V8 patch 0052 keeps functions created by `v8::Function::New()` in the +context snapshot so that their wrapper goes through the same path. Neither may throw: see the contracts on `Object::jsgSnapshotStateless` (`jsg.h`) and `Wrappable::snapshotClone()` (`wrappable.h`). A type with neither makes every Worker whose top @@ -253,8 +264,7 @@ its base type. Two outcomes are possible for a Worker that qualifies: - **The Worker does not qualify.** `supportsStartupSnapshot()` says no: an inspector, a module - fallback service, a service-worker script, Wasm modules, Python, the new module registry, C++ - streams, or a binding + fallback service, a service-worker script, Wasm modules, Python, C++ streams, or a binding of a kind that cannot transfer (`cryptoKey`, `memoryCache`, `wrapped`, logged at INFO as `startup snapshot skipped: a binding of this kind cannot be transferred`), or a dynamically loaded Worker. No zygote runs. @@ -312,6 +322,8 @@ once. | `src/workerd/api/tests/startup-snapshot-listeners-test` | top-level `addEventListener()` across the snapshot: `once`, `passive`, an object listener's `handleEvent`, a replaced global `addEventListener`; `signal` fallback | | `src/workerd/api/tests/startup-snapshot-functions-test` | a top-level `createRequire()` function across the snapshot; an `AsyncLocalStorage.bind()` function falls back | | `src/workerd/api/tests/startup-snapshot-modules-test` | module identity across the snapshot (ESM, JSON, CJS, `node:buffer`) | +| `src/workerd/api/tests/startup-snapshot-new-module-registry-test` | the same with the new module registry: re-importing modules the top level imported (also under a query or a fragment, and `import()` from inside those) and builtins, importing one it never loaded, and `import.meta`, including an `import.meta.resolve` the top level kept | +| `src/workerd/api/tests/startup-snapshot-node-test` | Node.js API objects (`AsyncLocalStorage`, `node:diagnostics_channel`) across the snapshot | | `src/workerd/api/tests/startup-snapshot-wrappables-test` | live objects (Headers, URL, TextEncoder, TextDecoder, crypto, navigator, bindings) and the identity of `crypto`, `crypto.subtle` and `navigator`, GC and a FinalizationRegistry cleanup across the snapshot; unclonable, `URLSearchParams`, a mid-stream `TextDecoder`, `node:fs` and ineligible cases fall back | | `src/workerd/api/tests/startup-snapshot-bindings-test` | bindings captured at module scope (service stub, Durable Object namespace and ID) used at request time; an RPC method taken off a stub, `wrapped` and `cryptoKey` fall back | | `src/workerd/server/tests/startup-snapshot-console` | console output printed once; no zygote under an inspector; a top level that throws gets no snapshot and its startup error is reported once | diff --git a/src/workerd/api/tests/BUILD.bazel b/src/workerd/api/tests/BUILD.bazel index 4ab5ecece..b0e775a18 100644 --- a/src/workerd/api/tests/BUILD.bazel +++ b/src/workerd/api/tests/BUILD.bazel @@ -1318,8 +1318,8 @@ wd_test( data = ["url-test.js"], ) -# The @all-compat-flags variant would turn new_module_registry on, which keeps the Worker out of -# the startup-snapshot pipeline and so would quietly test only the plain-Worker half. +# The @all-compat-flags variant would turn the temporal flag on, which the cases without that flag +# expect to be off. wd_test( # Some 25 Workers, each started from a zygote and then from its snapshot: past the "medium" # timeout in loaded debug builds. @@ -1331,18 +1331,14 @@ wd_test( ) # Top-level event listeners across a startup snapshot. Only the @all-autogates variant runs the -# pipeline. The @all-compat-flags variant would turn new_module_registry on, which keeps the Worker -# out of the startup-snapshot pipeline and so would quietly test only the plain-Worker half. +# pipeline. wd_test( src = "startup-snapshot-listeners-test.wd-test", args = ["--experimental"], data = ["startup-snapshot-listeners-test.js"], - generate_all_compat_flags_variant = False, ) # Module identity across a startup snapshot. Only the @all-autogates variant runs the pipeline. -# The @all-compat-flags variant would turn new_module_registry on, which keeps the Worker out of -# the startup-snapshot pipeline and so would quietly test only the plain-Worker half. wd_test( src = "startup-snapshot-modules-test.wd-test", args = ["--experimental"], @@ -1350,13 +1346,11 @@ wd_test( "startup-snapshot-modules-cjs-test.js", "startup-snapshot-modules-test.js", ], - generate_all_compat_flags_variant = False, ) # Live API objects (Headers, URL, TextEncoder, crypto, navigator, node:fs, bindings) across a # startup snapshot, and the cases that must fall back. Only the @all-autogates variant runs the -# pipeline. The @all-compat-flags variant would turn new_module_registry on, which keeps the Worker -# out of the startup-snapshot pipeline and so would quietly test only the plain-Worker half. +# pipeline. wd_test( src = "startup-snapshot-wrappables-test.wd-test", args = ["--experimental"], @@ -1364,23 +1358,19 @@ wd_test( "startup-snapshot-wrappables-fs-test.js", "startup-snapshot-wrappables-test.js", ], - generate_all_compat_flags_variant = False, ) # Capability bindings captured at module scope and used at request time, and the kinds that keep -# a Worker out of the pipeline. Only the @all-autogates variant runs the pipeline. The -# @all-compat-flags variant would turn new_module_registry on, which keeps the Worker out of the -# startup-snapshot pipeline and so would quietly test only the plain-Worker half. +# a Worker out of the pipeline. Only the @all-autogates variant runs the pipeline. wd_test( src = "startup-snapshot-bindings-test.wd-test", args = ["--experimental"], data = ["startup-snapshot-bindings-test.js"], - generate_all_compat_flags_variant = False, ) # Functions implemented in C++ across a startup snapshot. Only the @all-autogates variant runs the -# pipeline. The @all-compat-flags variant would turn new_module_registry on, which keeps the -# Worker out of the startup-snapshot pipeline and so would quietly test only the plain-Worker half. +# pipeline. The @all-compat-flags variant would turn new_module_registry on, under which the +# function createRequire('/worker.js') returns does not find ./lib.cjs. wd_test( src = "startup-snapshot-functions-test.wd-test", args = ["--experimental"], @@ -1391,10 +1381,19 @@ wd_test( generate_all_compat_flags_variant = False, ) +# Module identity and import.meta across a startup snapshot with the new module registry. Only +# the @all-autogates variant runs the pipeline. +wd_test( + src = "startup-snapshot-new-module-registry-test.wd-test", + args = ["--experimental"], + data = [ + "startup-snapshot-new-module-registry-case-test.js", + "startup-snapshot-new-module-registry-test.js", + ], +) + # Node.js API objects (AsyncLocalStorage, node:diagnostics_channel) across a startup snapshot. -# Only the @all-autogates variant runs the pipeline. The @all-compat-flags variant would turn -# new_module_registry on, which keeps the Worker out of the startup-snapshot pipeline and so would -# quietly test only the plain-Worker half. +# Only the @all-autogates variant runs the pipeline. wd_test( src = "startup-snapshot-node-test.wd-test", args = ["--experimental"], @@ -1402,7 +1401,6 @@ wd_test( "startup-snapshot-node-case-test.js", "startup-snapshot-node-test.js", ], - generate_all_compat_flags_variant = False, ) wd_test( diff --git a/src/workerd/api/tests/startup-snapshot-new-module-registry-case-test.js b/src/workerd/api/tests/startup-snapshot-new-module-registry-case-test.js new file mode 100644 index 000000000..102817559 --- /dev/null +++ b/src/workerd/api/tests/startup-snapshot-new-module-registry-case-test.js @@ -0,0 +1,63 @@ +// Copyright (c) 2026 Cloudflare, Inc. +// Licensed under the Apache 2.0 license found in the LICENSE file or at: +// https://opensource.org/licenses/Apache-2.0 + +// The case of startup-snapshot-new-module-registry-test.js. + +import { env } from 'cloudflare:workers'; +import unsafe from 'workerd:unsafe'; +import * as lib from './lib.js'; +import data from './data.json'; +// The registry records a resolution under its full specifier, query and fragment included, and +// restores it next to the definition it names (see IsolateModuleRegistry::restoreFromSnapshot()). +import * as variantQuery from './variant.js?q'; +import * as variantFragment from './variant.js#f'; +import * as buffer from 'node:buffer'; + +async function checks() { + const variantEvaluations = globalThis.__variantEvaluations; + const dynLib = await import('./lib.js'); + const dynData = await import('./data.json'); + const dynVariantQuery = await import('./variant.js?q'); + const dynVariantFragment = await import('./variant.js#f'); + const dynBuffer = await import('node:buffer'); + // V8 reports a module's canonical URL as the referrer of an import() in it, which for these is + // file:///bundle/variant.js; the registry maps that back to the specifier it was imported under. + const libFromVariantQuery = await variantQuery.load(); + const libFromVariantFragment = await variantFragment.load(); + const dynWorkers = await import('cloudflare:workers'); + // Not loaded at the top level: resolved, evaluated and given its import.meta at request time. + const lazy = await import('./lazy.js'); + return { + libNamespace: dynLib === lib, + libEvaluatedOnce: globalThis.__libEvaluations === 1, + jsonDefault: dynData.default === data, + variantQueryNamespace: dynVariantQuery === variantQuery, + variantFragmentNamespace: dynVariantFragment === variantFragment, + variantQueryUrl: variantQuery.url.endsWith('/variant.js?q'), + variantFragmentUrl: variantFragment.url.endsWith('/variant.js#f'), + // import.meta.resolve, taken at the top level, is a function the snapshot carries. + variantResolve: variantQuery.resolve('./lib.js').endsWith('/lib.js'), + variantQueryImports: libFromVariantQuery === lib, + variantFragmentImports: libFromVariantFragment === lib, + variantNotReEvaluated: + globalThis.__variantEvaluations === variantEvaluations, + builtinNamespace: dynBuffer === buffer, + workersEnv: dynWorkers.env === env, + lazyLinksToLib: lazy.token === lib.token, + lazyUrl: lazy.url.endsWith('/lazy.js'), + lazyNotMain: lazy.main === false, + // The main module's import.meta is first read here, at request time. + mainUrl: import.meta.url.endsWith('/worker.js'), + mainIsMain: import.meta.main === true, + }; +} + +export default { + async fetch() { + return Response.json({ + snapshotUsed: unsafe.startedFromStartupSnapshot(), + checks: await checks(), + }); + }, +}; diff --git a/src/workerd/api/tests/startup-snapshot-new-module-registry-test.js b/src/workerd/api/tests/startup-snapshot-new-module-registry-test.js new file mode 100644 index 000000000..388d0fad5 --- /dev/null +++ b/src/workerd/api/tests/startup-snapshot-new-module-registry-test.js @@ -0,0 +1,28 @@ +// Copyright (c) 2026 Cloudflare, Inc. +// Licensed under the Apache 2.0 license found in the LICENSE file or at: +// https://opensource.org/licenses/Apache-2.0 + +// Module identity and import.meta across a startup snapshot with the new module registry. The +// case Worker (startup-snapshot-new-module-registry-case-test.js) imports a bundle ES module, a +// JSON module, a module under a query and a fragment, and a builtin at the top level, then at +// request time imports them again, imports a module its top level never loaded, and reads +// import.meta. This runner asserts the report. Under the +// startup-snapshot autogate both services go through the zygote; without it they are plain +// Workers and the same assertions hold with snapshotUsed false. + +import { strictEqual } from 'node:assert'; +import unsafe from 'workerd:unsafe'; + +export default { + async test(ctrl, env) { + // The runner went through the same pipeline as the case: whether it was started from a + // snapshot says whether the pipeline is on in this variant. + const pipelineOn = unsafe.startedFromStartupSnapshot(); + const response = await env.case.fetch('http://case/'); + const r = await response.json(); + strictEqual(r.snapshotUsed, pipelineOn, 'case: snapshotUsed'); + for (const [check, value] of Object.entries(r.checks)) { + strictEqual(value, true, `case: ${check}`); + } + }, +}; diff --git a/src/workerd/api/tests/startup-snapshot-new-module-registry-test.wd-test b/src/workerd/api/tests/startup-snapshot-new-module-registry-test.wd-test new file mode 100644 index 000000000..29ee09d75 --- /dev/null +++ b/src/workerd/api/tests/startup-snapshot-new-module-registry-test.wd-test @@ -0,0 +1,53 @@ +using Workerd = import "/workerd/workerd.capnp"; + +# Module identity and import.meta across a startup snapshot with the new module registry; see +# startup-snapshot-new-module-registry-test.js. The runner's test() fetches the case service and +# asserts its report. The @all-autogates variant turns the startup-snapshot autogate on, so both +# services go through the zygote; the @ variant runs them as plain Workers. + +# The runner keeps the legacy registry, so that whether it was started from a snapshot says +# whether the pipeline is on, whatever happens to the case. +const runnerCompatibilityFlags :List(Text) = [ + "nodejs_compat", + "typescript_implemented_streams", + "experimental", + "unsafe_module", +]; + +const compatibilityFlags :List(Text) = [ + "nodejs_compat", + "typescript_implemented_streams", + "new_module_registry", + "experimental", + "unsafe_module", +]; + +const unitTests :Workerd.Config = ( + services = [ + ( name = "runner", + worker = ( + modules = [ + (name = "worker.js", esModule = embed "startup-snapshot-new-module-registry-test.js"), + ], + compatibilityFlags = .runnerCompatibilityFlags, + bindings = [ + (name = "case", service = "case"), + ], + ) + ), + ( name = "case", + worker = ( + modules = [ + (name = "worker.js", esModule = embed "startup-snapshot-new-module-registry-case-test.js"), + (name = "lib.js", esModule = "globalThis.__libEvaluations = (globalThis.__libEvaluations ?? 0) + 1; export const token = {};"), + (name = "lazy.js", esModule = "export { token } from './lib.js'; export const url = import.meta.url; export const main = import.meta.main;"), + (name = "variant.js", esModule = "globalThis.__variantEvaluations = (globalThis.__variantEvaluations ?? 0) + 1; export const url = import.meta.url; export const resolve = import.meta.resolve; export const load = () => import('./lib.js');"), + (name = "data.json", json = "{\"answer\": 42}"), + ], + compatibilityFlags = .compatibilityFlags, + ) + ), + ], + # typescript_implemented_streams needs the per-isolate bootstrap in every variant. + autogates = ["workerd-autogate-per-isolate-javascript-bootstrap"], +); diff --git a/src/workerd/io/worker.c++ b/src/workerd/io/worker.c++ index dbd28ec2a..b46026b08 100644 --- a/src/workerd/io/worker.c++ +++ b/src/workerd/io/worker.c++ @@ -1251,6 +1251,8 @@ Worker::Isolate::Isolate(kj::Own apiParam, lock->v8Isolate, reinterpret_cast(m.callback)); } jsg::isolateRegisterExternalReference(lock->v8Isolate, jsg::getSyntheticModuleEvalRef()); + jsg::isolateRegisterExternalReference( + lock->v8Isolate, jsg::modules::getSyntheticModuleEvaluationStepsRef()); configureJsgIsolate(*lock, features); diff --git a/src/workerd/jsg/modules-new.c++ b/src/workerd/jsg/modules-new.c++ index 91beb552d..153aa9821 100644 --- a/src/workerd/jsg/modules-new.c++ +++ b/src/workerd/jsg/modules-new.c++ @@ -384,6 +384,11 @@ class SyntheticModule final: public Module { std::span>(exports.data(), exports.size()), evaluationSteps); } + // See getSyntheticModuleEvaluationStepsRef(). + static intptr_t evaluationStepsRef() { + return reinterpret_cast(&evaluationSteps); + } + private: static v8::MaybeLocal evaluationSteps( v8::Local context, v8::Local module); @@ -1083,6 +1088,87 @@ class IsolateModuleRegistry final { return inner.getBundleBase(); } + // See recordIsolateModuleRegistryForSnapshot(). + kj::Array recordForSnapshot( + v8::Local context, v8::SnapshotCreator& creator) { + auto isolate = v8::Isolate::GetCurrent(); + kj::HashMap dataIndices; + kj::Vector records(resolutions.size()); + for (auto& resolution: resolutions) { + // Every resolution has its instantiation row (see resolveWithCaching). + Entry& entry = KJ_ASSERT_NONNULL(instantiations.find>( + resolution.key.id, resolution.value)); + size_t dataIndex = dataIndices.findOrCreate(&entry, [&]() { + return kj::HashMap::Entry{ + &entry, creator.AddData(context, entry.key.getHandle(isolate))}; + }); + records.add(SnapshotArtifact::NewModuleRegistryRecord{ + .contextType = static_cast(resolution.key.type), + .specifier = kj::str(resolution.key.id.getHref()), + .moduleDataIndex = dataIndex, + }); + } + // The v8::Globals of the modules, which CreateBlob() refuses. Nothing resolves in the zygote + // after this. + instantiations.clear(); + return records.releaseAsArray(); + } + + // The inverse of recordForSnapshot(), for a registry attached to an isolate restored from the + // snapshot: takes each recorded v8::Module out of the blob and re-creates the resolution and + // instantiation rows it had in the zygote, so import.meta, the dynamic-import referrer lookup + // and repeated imports find the module the heap already holds rather than a fresh compilation. + void restoreFromSnapshot(Lock& js, + v8::Local context, + kj::ArrayPtr records) { + kj::HashMap> taken; + for (auto& record: records) { + Url id = KJ_ASSERT_NONNULL(Url::tryParse(record.specifier.asPtr()), + "snapshot records an unparseable module specifier", record.specifier); + Url definitionId = id.clone( + Url::EquivalenceOption::IGNORE_FRAGMENTS | Url::EquivalenceOption::IGNORE_SEARCH); + ResolveContext resolveContext{ + .type = static_cast(record.contextType), + .source = ResolveContext::Source::INTERNAL, + .normalizedSpecifier = id, + .referrerNormalizedSpecifier = id, + }; + ResolveContext definitionContext{ + .type = resolveContext.type, + .source = resolveContext.source, + .normalizedSpecifier = definitionId, + .referrerNormalizedSpecifier = definitionId, + }; + const Module& definition = + KJ_ASSERT_NONNULL(inner.lookup(definitionContext, noopResolveObserver), + "snapshot records a module the registry does not define", record.specifier); + + v8::Local module = taken.findOrCreate(record.moduleDataIndex, [&]() { + v8::Local module; + KJ_REQUIRE( + context->GetDataFromSnapshotOnce(record.moduleDataIndex).ToLocal(&module), + "snapshot does not hold the module it was recorded to hold", record.specifier); + return kj::HashMap>::Entry{record.moduleDataIndex, module}; + }); + + if (instantiations.find>(id, &definition) == kj::none) { + instantiations.insert(Entry{ + .key = HashableV8Ref(js.v8Isolate, module), + .id = id.clone(), + .module = definition, + }); + } + resolutions.upsert(SpecifierContext(resolveContext), &definition, + [](const Module*& existing, const Module* replacement) { + KJ_ASSERT(existing == replacement); + }); + if (id != definition.id()) { + redirectedCanonicalIds.upsert( + definition.id().clone(), id.clone(), [](Url& existing, Url&& replacement) {}); + } + } + } + private: const ModuleRegistry& inner; const CompilationObserver& observer; @@ -1253,6 +1339,59 @@ v8::MaybeLocal SyntheticModule::evaluationSteps( } } +// The import.meta.resolve(...) function is effectively a shortcut for +// new URL(specifier, import.meta.url).href. The idea is that it allows +// resolving import specifiers relative to the current modules base URL. +// Note that we do not validate that the resolved URL actually matches +// anything in the registry. A functor with a clone() rather than a lambda, so +// that a startup snapshot can carry the function of a module whose top level +// touched import.meta (jsg::WrappableFunctionImpl's snapshotClone()). +struct ImportMetaResolve { + kj::String href; + + ImportMetaResolve clone() const { + return {kj::str(href)}; + } + + JsValue operator()(Lock& js, const v8::FunctionCallbackInfo& args) const { + // Node.js and the HTML spec both require import.meta.resolve to be called + // with a specifier argument; calling it with none is a TypeError rather + // than silently resolving the string "undefined". + JSG_REQUIRE(args.Length() >= 1, TypeError, "import.meta.resolve requires a specifier argument"); + // Note that we intentionally use ToString here to coerce whatever value is given + // into a string or throw if it cannot be coerced. + auto specifier = js.toString(args[0]); + // If Node.js Compat mode is enabled, a bare specifier naming a Node.js + // built-in (e.g. "fs") — or any "node:"-prefixed specifier — must resolve + // to the canonical "node:" URL, exactly as import() and require() do. + // Without this, import.meta.resolve("fs") would incorrectly resolve the + // bare specifier against the module's base URL (e.g. "file:///bundle/fs"). + if (isNodeJsCompatEnabled(js)) { + KJ_IF_SOME(nodeSpec, checkNodeSpecifier(specifier)) { + specifier = kj::mv(nodeSpec); + } + } + KJ_IF_SOME(resolved, Url::tryParse(specifier.asPtr(), href.asArray())) { + // import.meta.resolve is specified to be equivalent to + // `new URL(specifier, import.meta.url).href` (see the comment above). + // The WHATWG URL parser already collapses dot segments (including the + // percent-encoded "%2e"/"%2E" forms) during tryParse, so we can return + // the resolved href directly. + // + // We must NOT apply NORMALIZE_PATH here: normalizePathEncoding + // percent-decodes every escape whose byte is not in the path + // percent-encode set (e.g. "%66" -> "f"), which is not what + // `new URL(...).href` produces. Doing so silently rewrites an + // already-encoded specifier and diverges from Node.js/HTML, which + // leave such percent-encoding intact. + return js.str(resolved.getHref()); + } + // Node.js/HTML import.meta.resolve throws when the specifier cannot be + // resolved to a URL; it must not return null. + js.throwException(js.typeError(kj::str("Invalid module specifier: ", specifier))); + } +}; + // Set up the special `import.meta` property for the module. void importMeta( v8::Local context, v8::Local module, v8::Local meta) { @@ -1307,51 +1446,7 @@ void importMeta( } } - // The import.meta.resolve(...) function is effectively a shortcut for - // new URL(specifier, import.meta.url).href. The idea is that it allows - // resolving import specifiers relative to the current modules base URL. - // Note that we do not validate that the resolved URL actually matches - // anything in the registry. - auto resolve = js.wrapReturningFunction(js.v8Context(), - [href = kj::mv(href)]( - Lock& js, const v8::FunctionCallbackInfo& args) -> JsValue { - // Node.js and the HTML spec both require import.meta.resolve to be called - // with a specifier argument; calling it with none is a TypeError rather - // than silently resolving the string "undefined". - JSG_REQUIRE( - args.Length() >= 1, TypeError, "import.meta.resolve requires a specifier argument"); - // Note that we intentionally use ToString here to coerce whatever value is given - // into a string or throw if it cannot be coerced. - auto specifier = js.toString(args[0]); - // If Node.js Compat mode is enabled, a bare specifier naming a Node.js - // built-in (e.g. "fs") — or any "node:"-prefixed specifier — must resolve - // to the canonical "node:" URL, exactly as import() and require() do. - // Without this, import.meta.resolve("fs") would incorrectly resolve the - // bare specifier against the module's base URL (e.g. "file:///bundle/fs"). - if (isNodeJsCompatEnabled(js)) { - KJ_IF_SOME(nodeSpec, checkNodeSpecifier(specifier)) { - specifier = kj::mv(nodeSpec); - } - } - KJ_IF_SOME(resolved, Url::tryParse(specifier.asPtr(), href)) { - // import.meta.resolve is specified to be equivalent to - // `new URL(specifier, import.meta.url).href` (see the comment above). - // The WHATWG URL parser already collapses dot segments (including the - // percent-encoded "%2e"/"%2E" forms) during tryParse, so we can return - // the resolved href directly. - // - // We must NOT apply NORMALIZE_PATH here: normalizePathEncoding - // percent-decodes every escape whose byte is not in the path - // percent-encode set (e.g. "%66" -> "f"), which is not what - // `new URL(...).href` produces. Doing so silently rewrites an - // already-encoded specifier and diverges from Node.js/HTML, which - // leave such percent-encoding intact. - return js.str(resolved.getHref()); - } - // Node.js/HTML import.meta.resolve throws when the specifier cannot be - // resolved to a URL; it must not return null. - js.throwException(js.typeError(kj::str("Invalid module specifier: ", specifier))); - }); + auto resolve = js.wrapReturningFunction(js.v8Context(), ImportMetaResolve{kj::str(href)}); if (meta->CreateDataProperty( js.v8Context(), v8::Local(js.strIntern("resolve"_kj)), resolve) @@ -2188,11 +2283,29 @@ kj::Maybe ModuleRegistry::evaluateImpl(jsg::Lock& js, return kj::none; } +intptr_t getSyntheticModuleEvaluationStepsRef() { + return SyntheticModule::evaluationStepsRef(); +} + +kj::Array recordIsolateModuleRegistryForSnapshot( + v8::Local context, v8::SnapshotCreator& creator) { + return KJ_ASSERT_NONNULL(jsg::getAlignedPointerFromEmbedderData( + context, jsg::ContextPointerSlot::MODULE_REGISTRY)) + .recordForSnapshot(context, creator); +} + kj::Own ModuleRegistry::attachToIsolate(Lock& js, const CompilationObserver& observer) const { // The IsolateModuleRegistry is attached to the isolate as an embedder data slot. // We have to keep it alive for the duration of the v8::Context so we return a // kj::Own and store that in the jsg::JsContext - return kj::heap(js, *this, observer); + auto registry = kj::heap(js, *this, observer); + if (js.isStartingFromSnapshot()) { + // The context holds the zygote's evaluated module graph; the registry must know those modules + // (see SnapshotArtifact::newModuleRegistryRecords) rather than compile the bundle a second time. + registry->restoreFromSnapshot(js, js.v8Context(), + IsolateBase::from(js.v8Isolate).finalizedSnapshotArtifact().newModuleRegistryRecords); + } + return kj::mv(registry); } kj::Maybe ModuleRegistry::tryFindInBundle( diff --git a/src/workerd/jsg/modules-new.h b/src/workerd/jsg/modules-new.h index 3c9316f32..07eb6b8d4 100644 --- a/src/workerd/jsg/modules-new.h +++ b/src/workerd/jsg/modules-new.h @@ -880,6 +880,20 @@ class ModuleRegistry final: public kj::AtomicRefcounted, public ModuleRegistryBa friend class Module::Evaluator; }; +// The address of the evaluation-steps callback every synthetic module of the new registry is +// created with. V8 stores it as a Foreign on the module, so a startup snapshot needs it in the +// external-reference table (jsg::isolateRegisterExternalReference), like the legacy registry's +// getSyntheticModuleEvalRef(). +intptr_t getSyntheticModuleEvaluationStepsRef(); + +// For jsg::IsolateBase::createSnapshotBlob: stores the v8::Module of every module that the +// registry bound to `context` instantiated in the snapshot with `creator.AddData(context, ...)`, +// and returns the records (see SnapshotArtifact::newModuleRegistryRecords) from which +// ModuleRegistry::attachToIsolate rebuilds the registry of an isolate restored from the blob. The +// registry forgets the modules, so it is unusable afterwards; the zygote is discarded anyway. +kj::Array recordIsolateModuleRegistryForSnapshot( + v8::Local context, v8::SnapshotCreator& creator); + constexpr ModuleRegistry::Builder::Options operator|( const ModuleRegistry::Builder::Options& a, const ModuleRegistry::Builder::Options& b) { return static_cast( diff --git a/src/workerd/jsg/setup.c++ b/src/workerd/jsg/setup.c++ index 043b77c9b..bf83a104b 100644 --- a/src/workerd/jsg/setup.c++ +++ b/src/workerd/jsg/setup.c++ @@ -10,6 +10,7 @@ #include "setup.h" #include "libplatform/libplatform.h" +#include "modules-new.h" #include #include @@ -607,10 +608,14 @@ void IsolateBase::bakeConstructorTemplatesIntoSnapshot() { } void IsolateBase::bakeModuleRegistryIntoSnapshot(v8::Global& contextHandle) { - KJ_REQUIRE(!usingNewModuleRegistry, "snapshot not yet supported with the new module registry"); auto& artifact = mutableSnapshotArtifact(); v8::HandleScope scope(ptr); auto context = contextHandle.Get(ptr); + if (usingNewModuleRegistry) { + artifact.newModuleRegistryRecords = + modules::recordIsolateModuleRegistryForSnapshot(context, *getSnapshotCreator()); + return; + } auto& moduleRegistry = KJ_ASSERT_NONNULL(getAlignedPointerFromEmbedderData( context, ContextPointerSlot::MODULE_REGISTRY)); moduleRegistry.visitEntriesForSnapshot([&](ModuleRegistry::SnapshotHandleRef ref) { diff --git a/src/workerd/jsg/setup.h b/src/workerd/jsg/setup.h index 1a4a0761e..3374a3e48 100644 --- a/src/workerd/jsg/setup.h +++ b/src/workerd/jsg/setup.h @@ -786,6 +786,10 @@ class IsolateBase { // including paired TracedReferences. Entries never instantiated in the zygote leaked no identity into the // baked heap, so they are not pinned and are recompiled fresh at LOAD. The registry is owned // by the context global's Wrappable. + // + // With the new module registry, records its resolutions in + // SnapshotArtifact::newModuleRegistryRecords instead (see + // modules::recordIsolateModuleRegistryForSnapshot()). void bakeModuleRegistryIntoSnapshot(v8::Global& contextHandle); // Bakes every materialized JSG_RESOURCE-type constructor template diff --git a/src/workerd/jsg/snapshot.h b/src/workerd/jsg/snapshot.h index a094f4941..71dd3e3dd 100644 --- a/src/workerd/jsg/snapshot.h +++ b/src/workerd/jsg/snapshot.h @@ -64,7 +64,14 @@ class Wrappable; // compiling them again (preserving identity between the baked main-module graph and runtime // import()/require()). Field types are raw ints to avoid coupling this header to modules.h: // `moduleType` is a jsg::ModuleRegistry::Type (capnp ModuleType), `handleKind` a -// jsg::ModuleRegistry::SnapshotHandleKind. +// jsg::ModuleRegistry::SnapshotHandleKind. Only the legacy module registry's. +// * `newModuleRegistryRecords` are the new module registry's resolutions +// (modules::IsolateModuleRegistry): one (context type, specifier) the zygote resolved -> +// context-level AddData index of the v8::Module it resolved to; several specifiers may name one +// module. A restored isolate rebuilds its registry from these (ModuleRegistry::attachToIsolate); +// without them the registry starts empty, so a later import compiles and evaluates a second +// instance of a module the baked graph holds, and V8's import.meta callback, which looks the +// module up in the registry, leaves import.meta.url and import.meta.main unset. // * `listenerRecords` are the global scope's event listeners the zygote's top level registered // via addEventListener(), in per-type registration order: for each, its type, its `once` and // `passive` options, and the context-level AddData indices of its identity object and of the @@ -109,6 +116,13 @@ struct SnapshotArtifact: public kj::AtomicRefcounted { }; kj::Vector moduleRecords; + struct NewModuleRegistryRecord { + uint8_t contextType; // A modules::ResolveContext::Type. + kj::String specifier; // The normalized specifier URL, query and fragment included. + size_t moduleDataIndex; + }; + kj::Array newModuleRegistryRecords; + // A listener the zygote's top level registered on the context global, re-registered as it was // on load: the identity removeEventListener() matches, and the callback and receiver its // registration derived (a handleEvent object's method is read once, at registration). diff --git a/src/workerd/server/factory/worker-factory.c++ b/src/workerd/server/factory/worker-factory.c++ index 3c000e4b7..adac4c8e8 100644 --- a/src/workerd/server/factory/worker-factory.c++ +++ b/src/workerd/server/factory/worker-factory.c++ @@ -1588,8 +1588,7 @@ bool supportsStartupSnapshot(kj::StringPtr name, const WorkerSource& source, kj::Maybe> globals) { if (!featureFlags.getTypeScriptImplementedStreams() || featureFlags.getPythonWorkers() || - featureFlags.getNewModuleRegistry() || source.variant.is() || - hasWasmModules(source)) { + source.variant.is() || hasWasmModules(source)) { return false; } KJ_IF_SOME(words, globals) { @@ -1756,7 +1755,11 @@ kj::Promise> compileWorker(const WorkerFactory& constFac auto zygoteScript = zygoteIsolate->newScript(impl->name, workerSource, IsolateObserver::StartType::COLD, SpanParent(nullptr), newWorkerFileSystem(kj::heap(), getBundleDirectory(workerSource)), zygoteErrors, - api::pyodide::ArtifactBundler::makeDisabledBundler()); + api::pyodide::ArtifactBundler::makeDisabledBundler(), + // The registry only holds the bundle's definitions; each isolate attaches its own + // instances (jsg::modules::ModuleRegistry::attachToIsolate()). + newModuleRegistry.map( + [](kj::Arc& registry) { return registry.addRef(); })); // As the Worker's, except that `ctx.exports` is not kept: a v8::Global outliving the // zygote isolate breaks snapshot creation. zygote =