0
Fork 0
mirror of https://github.com/TanStack/router.git synced 2026-11-16 19:38:07 +00:00
router/benchmarks
Manuel Schiller d118cde15c test(router): stabilize opt-in Link performance comparisons
Bound control navigation history with replace and assert that the measured anchors remain mounted. Share the exact setup, batch, and post-measurement assertions between normal Vitest benches and the paired comparison runner. Increase normal warm-up and measurement windows.

Add an opt-in stable runner with a fresh process per case/replica, separate router/app modules, one shared production React runtime, deterministic V8 random/hash seeds, alternating initialization order, and ABBA/BAAB fixed-work blocks. Record main-thread CPU, wall time, whole-process CPU, bundle hashes, and all raw blocks. Derive per-case 95% intervals from independent process replicas rather than correlated individual batches; require CPU and wall results to corroborate a direction.

Same-code controls exposed substantial noise in earlier methods. The selected control was centered near zero (client CPU -0.95%, SSR CPU -0.56%), with intervals still several percentage points wide. Keep unresolved measurements explicitly inconclusive instead of labeling them regressions.

Compared fixed runtime refs origin/main 28a5e4504e and bbaa7b3f29 across all 26 cases with four fresh-process replicas each: 16 cases support speedups, 10 remain inconclusive, and none support a slowdown in both metrics. Client middleware CPU: -5.95% [-13.89%, +2.73%]; SSR middleware: -16.06% [-31.65%, +3.09%]; SSR unique params: -5.08% [-14.97%, +5.96%]. These do not establish middleware or unique-param regressions.

No production package code or default CodSpeed workflow changes. Keep the complete raw matrix, calibration trials, snapshots, and detailed conclusions in uncommitted session artifacts and LOG.md.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
History 2026-09-10 00:00:58 +02:00
..
bundle-size chore: simplify bundle optimization commands (#8253) 2026-09-05 20:46:34 +02:00
client-nav test(router): stabilize opt-in Link performance comparisons 2026-09-10 00:00:58 +02:00
memory test: reduce memory benchmark variance and simplify the harness (#8260) 2026-09-06 13:11:27 +02:00
ssr test(router): add opt-in React Link performance cases 2026-09-10 00:00:58 +02:00
cpu-simulation.ts test: stabilize CPU simulation with isolated SSR workers and explicit V8 settings (#8248) 2026-09-05 20:54:53 +02:00
README.md test: stabilize CPU simulation with isolated SSR workers and explicit V8 settings (#8248) 2026-09-05 20:54:53 +02:00
tsconfig.json test: stabilize CPU simulation with isolated SSR workers and explicit V8 settings (#8248) 2026-09-05 20:54:53 +02:00

CPU simulation benchmarks

client-nav measures navigation in Node/jsdom; ssr measures server requests. Both suites run through the CodSpeed Vitest integration in CPU simulation mode.

Worker runtime

Every CPU benchmark project includes cpuSimulationExecArgv() from cpu-simulation.ts. It augments CodSpeed's worker flags only during CPU simulation, including the legacy instrumentation mode.

The pinned CodSpeed integration supplies --no-opt. In Node 24 this is an alias for --no-turbofan, so Maglev can still optimize and inline hot functions. The additional --no-maglev keeps the optimizing compiler out of the simulation. --always-sparkplug compiles baseline code on first use, so warmup does not leave functions approaching a later tier-up or batch-compilation threshold. The worker regression test checks that even a once-called function receives baseline code. The integration also omits the old --no-scavenge-task flag on Node 20 and newer, where V8 renamed it to --no-minor-gc-task. Supplying the renamed flag keeps minor GC tied to allocation pressure instead of scheduled foreground tasks. Both flags follow the fixes in CodSpeed's Node 24 support.

--initial-old-space-size=512 fixes the initial/minimum old-generation allocation budget at 512 MiB. Removing only this override after worker isolation left two Vue SSR scenarios with same-commit spreads above 2% in five full-suite repetitions. The paired default-heap comparison with stock incremental-marking tasks also exceeded 2%. Runtime comparisons and tested alternatives are recorded in PR #8248.

This is a GC-policy tradeoff. Minor collections and allocation work remain measured, and CodSpeed still explicitly collects before measurement. However, V8 can skip automatic full collections below this old-generation budget, so the benchmark can omit some eventual full-GC CPU cost from increased object retention. Repeatability does not establish representative production GC frequency. The flag does not preallocate 512 MiB; the existing 4096 MiB maximum remains. See V8's initial/minimum heap configuration.

The configuration preserves CodSpeed's incremental-marking task settings; the additional --no-incremental-marking-task from the initial stabilization proposal has been removed.

The flags are intentionally absent from ordinary Vitest timing and walltime runs.

Changing simulation compiler settings changes the measurement baseline. Compare repeatability within each configuration before interpreting performance changes between configurations as router improvements or regressions.

Worker isolation

Keep one SSR workload in each speed*.bench.ts file. These projects explicitly enable Vitest isolation; CodSpeed uses the forks pool, so every file receives a fresh worker. Multiple workloads in one file share the worker's runtime state, even though CodSpeed collects before each measurement. Later control cases remained noisy after the runtime flag corrections. Running the React plain dehydration and linked-CSS controls alone reduced their observed three-run spreads to 0.09% and 0.02% respectively.

Variant files keep their workload names, request counts, concurrency, payloads and assertions. CodSpeed includes the file path in a benchmark's identity, so moved variants start a new history. The first workload keeps speed.bench.ts; the additional variants have descriptive file names alongside it. Type-check includes cover all of these files.

Streaming workload

Deferred payloads in the SSR streaming scenario resolve after two task turns on Node. A real 1 ms timer lets elapsed host time decide which concurrent requests resolve together. Counted turns preserve the initial loading render and deferred payloads; the existing HTML assertions check both. These benchmarks execute in Node and use setImmediate directly.

Repeatability checks

Dispatch the existing Benchmarks workflow against an unchanged branch:

gh workflow run client-nav-benchmarks.yml --ref <branch>

Wait for each workflow run to finish before dispatching the next repetition at the same commit. The workflow also runs memory benchmarks; exclude those results when assessing CPU repeatability. Check that every repetition has all expected CPU results (currently 132), rather than inherited results from an earlier run. Compare each benchmark's minimum and maximum across repetitions; keep input data, navigation/request counts, builds, dependencies and Node version fixed when testing runtime configuration changes.