# AGENTS.md This file provides guidance to coding agents working in this repository. ## Working Principles 1. **Think before coding.** State assumptions; if multiple interpretations exist, present them — don't pick silently. If a simpler approach exists, say so. If something is unclear, stop and ask. 2. **Simplicity first.** Minimum code that solves the problem: no speculative features, no abstractions for single-use code, no unrequested configurability, no error handling for impossible scenarios. If 200 lines could be 50, rewrite. 3. **Surgical changes.** Don't "improve" adjacent code, comments, or formatting; don't refactor what isn't broken; match existing style. Remove only orphans YOUR changes created; mention (don't delete) pre-existing dead code. Every changed line should trace to the user's request. 4. **Goal-driven execution.** Turn tasks into verifiable goals ("fix the bug" → "write a failing test, make it pass"); for multi-step work state a brief plan with a verify step per item, then loop until verified. ## Project Overview Monoscope is an open-source observability platform (Haskell, GHC 9.12.2) that ingests logs, traces, and metrics via OpenTelemetry and stores them in S3-compatible storage via TimeFusion (custom time-series DB backed by PostgreSQL/TimescaleDB). Frontend is server-side rendered HTML (Lucid + HTMX) with TypeScript web components (Vite). ### Storage migration: PG + TF dual-write is temporary Telemetry is currently dual-written to both Postgres (`otel_logs_and_spans`) and TimeFusion. **The end-state is TimeFusion only** — Postgres is the legacy store and is being phased out as TF features reach parity. When designing changes that touch the dual-write path (e.g. `bulkInsertOtelLogsAndSpansTF`, ingestion durability, idempotency), bias decisions toward TF correctness and treat PG as best-effort. Don't invest in PG-side complexity (e.g. `ON CONFLICT` ceremony, PG-specific reconciliation jobs) that will be deleted with the migration. ## File Structure The repo is three cabal packages (see `cabal.project`): - **`monoscope`** (root) — the server: library + `monoscope-server` exe + test suites. - **`monoscope-shared`** (`shared/`) — DB-free modules the server and CLI both need: wire types, the KQL grammar, the HTTP effect, CLI formatting. **Nothing here may import postgresql-simple, hasql, librdkafka or grpc** — the CLI links it and must stay free of native dependencies. - **`monoscope-cli`** (`cli/`) — the `monoscope` binary. Depends on `monoscope-shared` only, **never on `lib:monoscope`**. `scripts/check-cli-linkage.sh` fails the release build if that slips. `monoscope-cli` appears in `lib-deps.yaml` purely so the `test-dev` target (which compiles `src/` and `test/integration/` as one GHCi unit) can build the CLI specs — the same convention the other test-only deps there follow. **Never `import CLI.*` from `src/`**: it would compile, and it would put the server's Brick/Vty-laden CLI back inside the server. Both non-root packages have hand-written `.cabal` files (hpack only generates the root one); add new modules to their `exposed-modules` by hand. **Build the CLI with `make cli-build`, not a bare `cabal build … --ghc-options=…`.** `monoscope-shared` is also loaded by the ghcid session behind `make live-reload`; rebuilding it at a different optimisation level swaps the dylib underneath GHCi and the next reload dies with `symbol not found in flat namespace` (restart the pane with `make tmux-live-reload` if you hit it). ``` monoscope/ ├── app/Main.hs # Entry point — calls Start.startApp ├── shared/src/ # monoscope-shared: DB-free, shared with the CLI │ ├── Pkg/Deriving.hs # pure DerivingVia wrappers + ToSchema (no DB) │ ├── Pkg/Parser/{Expr,Stats}.hs # KQL grammar │ ├── Pkg/CLIFormat.hs, Data/Effectful/Wreq.hs │ ├── Pages/Charts/Types.hs, Models/Telemetry/Schema.hs, Web/Wire.hs ├── cli/ # monoscope-cli │ ├── CLI/{Main,Commands,Core,Config,Resource,Validate}.hs │ ├── CLI/{Chart,Dashboard,LogView,Table,UI}.hs # terminal rendering │ ├── exe/Main.hs # thin wrapper passing Paths_monoscope_cli version │ └── test/Main.hs # renderer unit tests (make cli-test) ├── src/ │ ├── Start.hs # App bootstrap, OTel init, server startup │ ├── BackgroundJobs.hs # odd-jobs background job definitions │ ├── ProcessMessage.hs # Pub/Sub & Kafka message processing pipeline │ ├── Utils.hs # Shared utilities │ ├── System/ # Config.hs (EnvConfig), Server.hs (Warp), Types.hs (effect aliases), Logging.hs, Tracing.hs │ ├── Web/ # Routes.hs (Servant wiring), Auth.hs (auth/JWT/error rendering) │ ├── Pages/ # Server-rendered HTML (Lucid + HTMX): BodyWrapper (shell), Components, LogExplorer/, Charts/, Monitors/, Bots/, Endpoints/, Onboarding/, Dashboards, Settings, Reports, Replay, Anomalies, … │ ├── Models/ # DB models & queries: Projects/ (Projects.hs = User/Project/Session/billing, ProjectApiKeys, ProjectMembers, Dashboards, GitSync), Apis/ (Endpoints, Monitors, Anomalies, LogPatterns, Fields, Integrations, Issues/), Telemetry/ (Telemetry.hs queries, Schema, SummaryGenerator), Users/Sessions.hs (re-exports) │ ├── Pkg/ # Shared libs: DeriveUtils (DB types/newtype/TH; re-exports Pkg.Deriving), Parser (KQL, Megaparsec), AI, Mail, Queue (Pub/Sub+Kafka), QueryCache, TestUtils, Drain, ErrorFingerprint, PatternMerge, GitHub, EmailTemplates, Components/, Parser/ │ ├── Opentelemetry/ # OtlpServer.hs (gRPC OTLP, port 4317), OtlpMockValues.hs │ └── Data/Effectful/ # Custom effects: LLM, Notify, UUID, Wreq ├── test/ # doctests/, unit/ (no DB), integration/ (needs DB), bench/, golden/ ├── static/ # migrations/ (numbered SQL), public/ (CSS/JS/images/fonts) ├── web-components/src/ # TypeScript frontend (Vite): web components, chart-cli, workers ├── proto/opentelemetry/ # OpenTelemetry .proto files ├── package.yaml # Hpack build definition (→ monoscope.cabal) ├── Makefile # Dev commands (build, test, lint, fmt, …) └── docker-compose.yml # TimescaleDB + app services ``` ## Development Workflow A `make live-reload` process is always running in a tmux pane, logging to `build.log`. Web component build logs are in `web-components.log` and CSS build logs are in `css.log`. ### Keep the watchers running — never restart them to iterate **Iterate against long-lived ghcid watchers in tmux panes: `make tmux-live-reload` (server, `build.log`) and `make tmux-live-test-dev` (tests, `build-test-dev.log`).** Start each once and leave it running. After the first load, a save recompiles only the changed modules and reruns in seconds; every kill-and-restart reloads all ~220 modules and costs 5–12 minutes. On 2026-10-19 a single session lost hours to restarting `live-test-dev` once per test. - **Change which tests run by editing `.test-args`, not by restarting.** `echo '--match "LiveTail"' > .test-args` reruns the live watcher on that focus (measured: 9 s, against ~10 min for a restart). `TEST_MATCH=…` only seeds the file at startup. - **Never `tmux new-window`/`new-session` for a watcher or dependency.** Use the `make tmux-*` targets (`tmux-live-reload`, `tmux-live-test-dev`, `tmux-web-components-watch`, `tmux-minio-local`): one pane per worktree+job in the shared `services` window, reused from any window. - Restart a watcher only when it has crashed or wedged (see below), not to change focus. - **Live GHCi sessions are capped.** `live-reload` and `live-test-dev` refuse to start once `GHCI_MAX` (default 6) are running (`scripts/reap-ghci.sh`), since each holds 5–15 GB and too many hang Docker. Stop an idle watcher (Ctrl-C in its pane) rather than raising the cap. - `monoscope-shared` (`shared/`) is a separate package the test watcher loads prebuilt. Edits there are invisible to a running watcher, so that is the one case that needs a restart. - **`test/unit` is not in `live-test-dev`.** Run `make live-test-unit` in the background for unit specs. A one-shot `make test-unit` recompiles the whole library each time, about 10 minutes per run. - **Batch edits to widely imported modules.** Nearly every spec imports `Pkg.TestUtils`, so each save to it recompiles most of the suite, taking 1–5 minutes. Make all edits to such a module, then save once. - **One full-suite run per PR.** Iterate with a focused `.test-args`. Clear `.test-args` for the full suite (about 8 minutes) only once the change is complete. Changes to test infrastructure, such as clocks or fixtures, need that full run, and need it only once. - **Address review findings before `make ci-signoff`, not after.** A sign-off takes 15–25 minutes, and any fix after it means another full run. Run the review skills, apply their findings, then sign off once. A dashboard e2e test flakes locally. Before investigating, rerun it alone with `make ci-signoff CHECKS="e2e"`. - **Wait on the watcher by reading its newest block.** The `grep` on this machine is ugrep, which mis-handles some extended patterns. Ad-hoc `grep`/`until` loops have stalled until timeout or reported "compiled" while errors were present. Read the text after the last `Reloading` in `build-test-dev.log` with a small script, and check the result for `error`. ### Pull requests only All Monoscope changes must go through a pull request, including documentation, configuration, hotfixes, and deploy-related changes. Work on a non-`master` branch, push that branch, and iterate by updating the pull request until its review and required checks pass. Merge the pull request through GitHub; **never push commits directly to `origin/master`**. When a production deployment is needed, merge the pull request first and deploy the resulting commit already present on `origin/master`. Do not use the deploy workflow to bypass pull-request review. ### DO NOT run `cabal build` to verify your changes **To check your changes compile, read `build.log`. Do not run `cabal build`, `cabal repl`, `ghc`, or any equivalent yourself.** ghcid recompiles automatically on every file save and streams the result into `build.log`; running `cabal build` in parallel races it, warms a second build cache, and wastes minutes of your time (and the user's) for information that is already sitting in the log file. This applies to **every** edit, including the last one you just made. If you caught yourself about to run `cabal build`, stop and `Read` `build.log` instead. **If `build.log` shows ghcid has crashed** (e.g. `cabal: semWait: invalid argument`, an empty tail, or a hang), restart the tmux pane — do **not** fall back to `cabal build`: ```bash make tmux-live-reload ``` Then re-read `build.log` after a few seconds. The only times it's appropriate to invoke the compiler directly are: running the test suite (`make test-*`), running doctests, or running `hpack` after adding/removing `.hs` files. ## Build Commands ```bash # Build (fast, dev) cabal build lib:monoscope --ghc-options="-O0 -j8" # Build all (lib + exe + tests) cabal build all -j --ghc-options="-O0 +RTS -A64m -n2m -RTS" # Run server cabal run # Live reload (ghcid) make live-reload # Type-check watch (ghciwatch) make watch # Format make fmt # fourmolu on src/**/*.hs # Lint make lint # hlint src make fix-lint # hlint auto-refactor # CSS (Tailwind v4 + DaisyUI v5) make post-css # one-shot build make css-start # watch mode (logs to css.log) # Web components (Vite) make web-components-watch # watch mode (logs to web-components.log) ``` **NEVER run `cabal clean`** — it destroys all cached deps and dramatically increases compile time. **Never put a `package *` stanza in `cabal.project.local`.** Its `ghc-options` feed every dependency's `~/.cabal/store` hash, forking a private copy of the whole dependency set per checkout — that is what used to make each worktree rebuild everything from scratch. Scope dev flags to the named local packages (`package monoscope` / `monoscope-shared` / `monoscope-cli`) instead. New worktrees: `make worktree NAME=my-feature` (see the Makefile comment for what it does and does not seed). After adding/removing `.hs` files, always run `hpack` to regenerate `monoscope.cabal` from `package.yaml`. ### Dead Code Detection (Weeder) Run `weeder` regularly to find unused functions and remove them. A `weeder.toml` config is in the repo root. Ignore results from these files (they contain intentionally unused/future-use code): - `src/Data/Effectful/Wreq.hs` — HTTP method effect wrappers (used as needed) - `src/System/Logging.hs` — logging helpers - `src/System/Tracing.hs` — tracing helpers - `src/Devel.hs` — dev-mode REPL helpers - Auto-generated proto-lens files (`dist-newstyle/build/autogen/Proto/`) - `cli/CLI/` — CLI tool (separate executable, weeder may not trace its root) ## CI, and running it locally Use standard GitHub-hosted runners for remote jobs. Do not add Blacksmith runners or actions without an explicit user request. ### Run the reviews before updating the pull request Before pushing a pull-request branch, run all three over the change and **implement what they raise**: ``` /hs-distill # size, reuse, algebraic style /hs-evasion-review # constraint-evading compromises /hs-lob-review # client-side tier & locality of behavior ``` Then **run them again on the result**, and keep looping until a pass comes back with nothing material left — a fix routinely opens the next finding, so one round is not a sign-off. Once that loop has settled, run `/simplify` as the last pass over the final diff, and apply what it finds. Then `make ci-signoff`, push the branch, and update the pull request. Not every finding has to be taken: if one is wrong or the cost outweighs it, say so explicitly and move on. What is not acceptable is updating the pull request without having run them. ### Build production images on this Mac **Generate production images on this Mac with OrbStack.** Use `linux/amd64` emulation through Rosetta. Do not use production servers, SSH builders, or GitHub runners to generate these images. A command started on this Mac can still compile on a remote host. Inspect the Docker context and builder endpoint before an image build. The endpoint must use the local `orbstack` context, whose Docker socket is local to this Mac. Set `DOCKER_CONTEXT=orbstack` and `MONOSCOPE_BUILDER=monoscope-local-amd64` explicitly for `ci-signoff`, `deploy-image`, and `ship`. Image generation checks the Mac host, local Unix socket, builder name, and sole OrbStack node; missing or remote builders cannot fall back. GitHub can reuse a completed image, but a cache miss fails instead of compiling there. `builder-setup` creates only a local node and rejects `BUILD_HOST`. See the procedure in `docs/local-ci.md`. ### Sign off locally before updating the pull request — never push and wait on CI **Run sign-off only at the end, after all requested work, reviews, and focused verification are complete. Sign-off is expensive: do not start or queue `make ci-signoff`, full CI, or deploy-image builds while implementation or UI iteration is still in progress. Use the existing watchers and focused tests during development. If more work is requested after sign-off starts, stop that run and finish the work before starting the final sign-off again.** **A push to `master` IS a deploy, and remote CI is far too slow to be a feedback loop: an image build alone runs ~85 minutes.** So `make ci-signoff` must be green *before* you push the branch. It runs the checks locally and publishes attestations, which CI's gate then reuses — a signed-off pull-request update can reuse those results instead of re-running everything. ```bash DOCKER_CONTEXT=orbstack MONOSCOPE_BUILDER=monoscope-local-amd64 make ci-signoff DOCKER_CONTEXT=orbstack MONOSCOPE_BUILDER=monoscope-local-amd64 make ci-signoff CHECKS="fmt lint" ``` This is a rule, not a preference: on 2026-09-08 five consecutive deploys failed, each burning ~85 minutes of wall clock before reporting an e2e expectation that a local run would have shown in minutes. Never treat "push it and see what CI says" as a way to test a change. **Sign-off must stay fast, every time — treat a slower sign-off as a regression.** The cabal store, `dist-newstyle`, npm and Playwright caches are docker volumes shared by every checkout (`ci/compose.yml`), so a new worktree or sign-off clone builds incrementally. Never key a cache per checkout: that leaked ~400GB of volumes and made every clone a cold build. If one check misbehaves on shared state, fix that check — never slow every check down for it. The precedent: after another branch built into the shared `dist`, GHC left some live modules without `.hie` files and weeder reported their callees dead, so the weeder check recompiles any tracked module missing one. **`make land PR=` merges without losing the race to a busy master**: it signs off, waits for the tree's image, and merges only while `origin/master` is still the base; if master moved, it rebases and repeats. Merging after master moved leaves a tree with no image, so the deploy fails. **`make ship` is the whole deploy from here** — checks, image, push, CapRover — so the slow half stops being a queue you wait on: ```bash DOCKER_CONTEXT=orbstack MONOSCOPE_BUILDER=monoscope-local-amd64 make ship make deploy-status # what production is running right now ``` It refuses to deploy a commit `origin/master` does not contain, and refuses when any deploy-path check (`build doctests unit-tests cli-tests integration-tests e2e`) lacks an attestation for that exact tree. CI still runs, but the gate skips every attested check, the image is already in the registry, and the deploy job asks `ci.sh deployed ` and skips too. Use the local OrbStack builder for image generation, as required above. Production hosts receive completed images only when deployment is authorized. Two things this does **not** cover, so check them yourself: - **The image build is outside the fingerprinted checks** (it produces an artifact, not a verdict), so `ci-signoff` cannot prove it. When a change touches the `Dockerfile`, `static/`, or the web-components build, run `DOCKER_CONTEXT=orbstack MONOSCOPE_BUILDER=monoscope-local-amd64 make deploy-image` first — that is the only local proof of the stage that has failed most often. - **If a check genuinely cannot run here** (missing service or tool), say so in the PR/commit and let CI run that one. Never publish an attestation for a check that did not pass. `ci/checks.tsv` is the single definition of what CI is; `.github/workflows/*.yml` and `scripts/ci/ci.sh` both read it, so a check cannot exist in one and not the other. Each check is fingerprinted over the content it depends on, and a pass publishes an attestation under `refs/ci-attest/v1/*`. CI's gate job skips any check already proven for the exact tree it is about to test — by an earlier run, another branch, or a developer's `make ci`. ```bash make ci # run CI's checks in CI's own containers, attesting each pass make ci-status # what CI would run right now, without running any of it make ci-selftest # fingerprint/capability logic + checks.tsv <-> run_body agreement ``` **Adding or changing a check means editing `ci/checks.tsv` and `run_body` in `scripts/ci/ci.sh` together** — `make ci-selftest` enforces they agree, and also that every declared input path exists (a typo silently narrows what a check depends on, which is how an untested change ships). Things that are load-bearing and easy to break: - **Narrow a check's `inputs` only where it is provably sound.** Too wide costs a rerun; too narrow ships an untested change. The frontend bundles are in the Haskell pathset on purpose — BodyWrapper TH-splices their content hash. - **Capabilities are the safety property.** A check declares what it needs (`ghc`, `pg`, `minio`, `tf-real`); an attestation records what the environment had. Reuse requires provided ⊇ required, so a stub-service run can never satisfy a check needing the real one. Never widen a `caps` string to make something pass. - **Fingerprints are pinned once per run**, because steps rewrite the tree as they go (`hpack`, `npm ci`, `hlint --refactor`). Recomputing after those would attest a fingerprint no gate ever asked for. - **Never let bookkeeping fail a green check.** Attestation publishing is best-effort; a network or git failure must not turn a passing suite red. This already happened once — `set -e` plus git's container-job "dubious ownership" reported a successful 16-minute build as a failure. - **`docs/local-ci.md` is the reference** — every knob, the fidelity caveats, and the kill switches (`CI_ATTEST_DISABLED` repo variable, `EPOCH` in `ci.sh`). The deploy image is deliberately **not** fingerprinted: it produces an artifact rather than a verdict, so its cache is the registry and its identity is the commit SHA. `build-image` skips when `ghcr.io/…:` already exists, and `make deploy-image` lets a big machine pre-build it. Because an image cannot be re-derived from source to check (Haskell builds are not bit-reproducible), that path refuses a dirty tree and records who built it; the deploy job prints `built by: …` every time. ## Testing ### NEVER run `cabal test` / `stack test` to verify tests **Read `build-test-dev.log`.** `make live-test-dev` is the way to run and iterate on the test suite in this repo. It runs as a ghcid watcher (in a tmux pane or as a backgrounded process), compiles `src/` + `test/integration/` as a single GHCi target (no library re-link between edits), and re-runs the suite automatically on every file save, streaming results into `build-test-dev.log`. This is the **same** rule as `cabal build` vs `build.log` — running `cabal test` in parallel re-links the whole library, races the watcher, warms a second build cache, and wastes minutes of your time and the user's for information that's already in the log file. ```bash make tmux-live-test-dev # start once, in its own tmux pane; auto-reruns on save echo '--match "/MonitoringSpec/"' > .test-args # refocus the running watcher (no restart) : > .test-args # back to the full suite ``` If you catch yourself about to run `cabal test integration-tests` or `cabal test --test-options=...`, **stop and read `build-test-dev.log` instead.** It applies to every test verification, including the one you just decided was "quick" or "scoped." If the watcher isn't running, start it with `make tmux-live-test-dev` (or `make live-test-dev` in the background); do **not** fall back to a one-shot `cabal test`. If the watcher seems stuck or crashed, restart it the same way — never invoke the compiler directly. The one acceptable exception is a CI-style reproduction with a fixed seed (e.g. chasing a flake) — and even then, prefer rerunning under live-test-dev, focused through `.test-args`, first. ### Other test commands (rarely needed) ```bash make test # all tests (requires DB) — heavy; only for full CI parity make cli-test # CLI renderer tests (no DB, milliseconds) make test-unit # unit tests (no DB) make test-doctests # doctests make test-integration # integration tests (requires DB) # Live watching for non-integration targets make live-test-unit make live-reload-doctests ``` The `test-dev` stanza in `package.yaml` shares its dependency list with the library via `!include hpack-includes/lib-deps.yaml` — when you add a library dep, edit that file and both targets pick it up. (hpack can't splice list-position includes, so a few test-only deps — hspec, uuid-quasi, ki, pg-transact, aeson-pretty, process — also live in `lib-deps.yaml`; the library declares them unused, silenced by the global `-Wno-error=unused-packages`.) **The container runtime on macOS is [OrbStack](https://docs.orbstack.dev/quick-start), not Docker Desktop.** It supplies the `docker`/`docker-compose` CLIs and its own `orbstack` docker context, so every `docker …` command and `make` target in this file works unchanged — but Docker Desktop is uninstalled, so anything that assumes `/Applications/Docker.app`, the `desktop-linux` context, or the `desktop` credential helper will fail. If `docker` can't reach a daemon, start OrbStack (`open -a OrbStack`) rather than reinstalling Docker Desktop. Integration tests require PostgreSQL+TimescaleDB at localhost:5432 (postgres/postgres). Start with: ```bash make timescaledb-docker # persistent make timescaledb-docker-tmp # ephemeral (tmpfs) ``` **`USE_EXTERNAL_DB=true` is what points the suite at that container — never drop it.** Every DB-touching make target sets it (`test`, `test-integration`, `test-integration-tf`, `live-test-dev`). Without it, `Pkg.TestUtils.withSetup` takes the `withLocalSetup` branch and boots **tmp-postgres** from whatever `postgres` binaries are on `PATH` — on a Mac that is homebrew's, which ships `timescaledb` but **not** `timescaledb_toolkit`. Migration `0001` then fails on `CREATE EXTENSION IF NOT EXISTS timescaledb_toolkit`, the template database never builds, and **every** example dies in setup with `uncaught exception: SomePostgreSqlException`. That failure looks exactly like a broken change. Tells that it is the flag instead: - the error names an `/opt/homebrew/...` extension control file, and the Postgres log lines say `host=[local]` (a Unix socket — the container is reached over TCP) - a spec your change did not touch fails identically - `DB_HOST` makes no difference; it only applies to the external-DB branch So: run the make target. If you invoke `cabal test integration-tests` directly, pass `USE_EXTERNAL_DB=true` yourself — and before blaming a change for integration failures, run one spec it did not touch. **Golden files** (`test/golden/`): integration tests cache external API responses; `UPDATE_GOLDEN=true` re-records them with **real outbound HTTP/LLM calls**. HTML/text/JSON snapshots (`goldenText`/`goldenHtml`) regenerate with `UPDATE_SNAPSHOTS=true` instead — never set `UPDATE_GOLDEN` to refresh a snapshot. **Testing is critical — always write tests for new features and bug fixes.** Prefer high-level e2e integration tests that exercise handlers or real user actions end-to-end (from HTTP request through DB and back). Tests should assert on responses as users would see them. Only use doctests for isolated pure functions. Avoid low-level unit tests that mock internals — test real behavior through the actual stack. Keep tests concise: no redundant tests, prefer single tests that cover multiple cases implicitly or explicitly rather than many narrow tests. When you doctest a smart constructor or refined type, **doctest the invariant, not just a happy path** — pin both the round-trip and the rejection (e.g. `mkX bad == Left …`) so the guarantee the type promises can't silently regress. **Bug-fix workflow (mandatory):** 1. **Reproduce the bug as a failing test first** — at the level closest to where the bug manifests (integration if it's a handler/DB issue, doctest if it's a pure function). The test must fail in a way that exposes the bug, not just "something errored." 2. **Then write the fix** — and confirm the test now passes. 3. **Keep the test in the suite as a regression guard.** Name it after the bug (e.g. `splitReplayPayload_emptyStringFields_doesNotDropMessage`) so future readers see what it's preventing. Reference the incident/issue in the test if relevant. 4. Skipping the failing-test step is **not** acceptable just because the fix "looks obvious" — the day's experience is full of "obvious" fixes that didn't actually cover the real path (e.g. fixing simple-query ABORT while Hasql uses extended-query). A failing test forces you to exercise the actual code path. Test helpers in `Pkg.TestUtils`: `withTestResources` (pool, caches, auth), `toServantResponse` (for handler responses), `runQueryEffect` (for DB effects), `ingestLog`/`ingestTrace`/`ingestMetric` (OTLP ingestion). `Pkg.TestUtils` is also the test prelude: it re-exports Relude, the Hspec vocabulary, `sql`, `Only`, `def` and `UUIDId`, so specs import it instead of those. Fixture SQL goes through `dbExec`/`dbQuery`/`dbOne` (on the test clock), handler payloads through `call`, presentation through `goldenHtml`/`goldenText` (`UPDATE_SNAPSHOTS=true` rewrites; `UPDATE_GOLDEN=true` instead re-records external HTTP/LLM responses). - project is a demo project which can be accessed with no auth. ## Architecture ### Effect System Uses **effectful** for algebraic effects. Key types in `System.Types`: - `ATBaseCtx` — base effect stack (no auth) - `ATAuthCtx` — authenticated request context - `DB es` — DB access constraint (`WithConnection :> es, IOE :> es`) ### Key Modules - `Start.hs` — app bootstrap + OpenTelemetry init - `Web/Routes.hs` — Servant route definitions - `Web/Auth.hs` — auth + cookie/JWT - `Pages/` — server-side HTML pages (Lucid + HTMX) - `Models/Projects/Projects.hs` — User, Project, Session models - `Pkg/Parser.hs` — KQL query parser (Megaparsec) - `Pkg/DeriveUtils.hs` — DB types, newtype helpers, TH utilities - `Pkg/TestUtils.hs` — test infrastructure - `Opentelemetry/OtlpServer.hs` — gRPC OTLP ingestion (port 4317) - `ProcessMessage.hs` — message processing pipeline - `BackgroundJobs.hs` — odd-jobs background jobs ### Ingestion Pipeline Data enters via gRPC OTLP (port 4317), Google Pub/Sub, or Kafka → `ProcessMessage.processMessages` ### Database - Primary: PostgreSQL + TimescaleDB (pg-entity + postgresql-simple) - Secondary: TimeFusion (S3-backed, PostgreSQL wire protocol) - Migrations: `static/migrations/` (numbered, run on startup when `MIGRATE_AND_INITIALIZE_ON_START=True`) - **Migrations are append-only — never edit an applied migration.** The runner checksums each file; changing one that already ran causes a checksum mismatch that silently halts the runner mid-list, so every later migration never applies. To change schema, add a new numbered follow-up migration. - **Ordering is the filename, not the number — check the highest number on `master` before claiming one.** Two branches that both add `0125_*` merge cleanly and both apply, in `sort` order of the full filename (`0125_git_hosts.sql` before `0125_issue_ack_window.sql`). `0125` is already taken twice for this reason. It is only a naming hazard, not a live bug, and it is **not** fixable after the fact: renaming an applied file changes the checksum and halts the runner. ### Frontend - Lucid (type-safe HTML) + HTMX for interactivity - TypeScript web components in `web-components/` (Vite) - Static assets in `static/public/assets/` ## Type & Effect Design The type system is the primary tool for correctness here. Lean on it before writing runtime checks or tests. - **Make invalid states unrepresentable.** Model the business domain in types so illegal combinations can't be constructed. Prefer sum types over boolean/`Maybe` soup (`data PlanState = Trial UTCTime | Active SubId | Cancelled UTCTime` over `isTrial :: Bool` + `subId :: Maybe SubId` + `cancelledAt :: Maybe UTCTime`). Push invariants into constructors via smart constructors + hidden data constructors (export the type, not its constructor; expose a `mkX :: … -> Either Err X` that validates once). If a function has to defend against a state the caller "shouldn't" produce, that state shouldn't be representable — fix the type, delete the guard. - **Parse, don't validate.** Validate at the boundary (request decode, env parse, DB read) into a precise type, then trust that type everywhere downstream. A `NonEmpty`, a `ProjectId` newtype, or a refined `Email` carries its guarantee through the whole call graph — no re-checking, no "this is always non-empty here" comments. Use `newtype` aggressively to stop primitive obsession (`UUIDId`-style typed IDs are the existing pattern — extend it: don't pass bare `Text`/`Int` when a domain meaning exists). - **Encode invariants in types, not comments.** A comment saying "callers must hold the lock" / "list is sorted" / "only call after auth" is a type that wasn't written. Reach for phantom types, `data`-kind tags, GADTs, or a capability argument before a comment. Totality matters: avoid partial functions (`head`, `fromJust`, partial record fields, incomplete `case`) — `-Wincomplete-patterns` is on for a reason. - **Prefer `Eff` effects over raw `IO`.** A concrete `IO a` says nothing about what a function does; an `Eff '[DB, LLM, Time, Log] a` signature documents and *constrains* its capabilities — a function with no `Notify` in its row provably can't send notifications. Program against the smallest effect row that does the job (`(DB :> es, Time :> es) => Eff es a`, not `ATAuthCtx`-everything). New side-effecting capabilities should become an effect in `Data/Effectful/` (see `LLM`, `Notify`, `UUID`, `Wreq`, and `Time`/`TestClock` from PR #329), not an `IO` call buried in a handler. - **`bracket`/`withX` for every acquired resource.** Connections, file handles, pool checkouts, temp state — acquire and release inside `bracket`/`finally`/a `withX` wrapper so an exception can't leak the resource. Never manual acquire-then-release in sequence; the ingestion pipeline and DB pools make a leaked handle a production incident, not a tidiness issue. - **One canonical interpreter pair per effect.** When you add an effect in `Data/Effectful/`, ship both interpreters in that same module: the production one and the test/fake one. "How do I fake this in a test?" should never require hunting — the deterministic interpreter lives next to the effect by construction. - **Effects make tests reproducible.** The reason to define an effect is the test interpreter: non-determinism (clock, UUID, randomness, HTTP, LLM) goes behind an effect so tests run a pure/fake interpreter and assert deterministically (`TestClock` for `NOW()`, golden interpreters for external APIs). If you find yourself wanting to mock or stub something in a test, that something should be an effect. Never reach for `IO`-level seams (global `IORef`, `unsafePerformIO`, env-var toggles) where an effect interpreter is the right seam. ## Conventions - **Client-side escalation ladder — lowest tier that stays clear wins.** Plain HTML → Tailwind CSS state variants (`group-has`, `peer-has`, `has()`, `:checked`) → HTML + concise declarative attributes (htmx, `hx-live` bindings) → hyperscript. **Hyperscript is a concise DSL that will always remain in this codebase where it is the clearest, most concise solution** — `send X to `, `halt`, class add/remove one-liners, cross-element listeners, keyboard handlers, behaviors. Never translate a hyperscript one-liner 1:1 into an `hx-on:*` raw-JS string (`document.getElementById(...)` chains); that swaps a readable DSL for wordier JS and is a regression. Replace hyperscript only when the replacement is strictly simpler and more concise: a CSS variant, a declarative binding, or JS that collapses multi-line DSL into one clear line. - **Locality of behavior.** Keep behavior at the site where it is used — inline hyperscript, event handlers, and short scripts directly on the element rather than naming them in a `where` clause or module-level binding. A reader should be able to read a button and immediately see what it does without jumping elsewhere. Named helpers are justified only when the same logic is reused across multiple call sites *and* the name adds clarity; single-use extractions trade readability for false DRY. This applies equally to HTML attributes, JS snippets, and HTMX directives: if it's only used once, keep it at the use site. - **Lucid owns HTML; HTMX owns HTML arrival.** Render UI markup in Lucid whenever possible — never construct HTML strings in JavaScript. For async content, render a stable Lucid shell/loading state, then have HTMX swap a dedicated child target with the response. JavaScript may only coordinate state or interop; it must not become a second rendering system. - **Conciseness is the top priority.** Minimize files and lines of code. Use advanced Haskell techniques (generics, `DerivingVia`, `DeriveAnyClass`, Template Haskell, type-level programming), GHC extensions, and libraries to eliminate boilerplate. Prefer editing existing code to cover new functionality over creating new files. Write general functions that solve multiple use cases. Advanced techniques are encouraged when they reduce code. Prefer fewer, larger files over many small ones. - **Be succinct and concise — in code, in comments, in PR descriptions, and in replies.** Keeping code small is a priority on its own. Drop helpers used once, prefer point-free where it stays readable, skip restating what the code already shows, and don't add prose where a one-line note suffices. Same applies to commit messages and review responses. - **Prefer external packages over in-house — always.** Reaching for a maintained library is the default, not the fallback: from one-line primitives (`BS.filter`, `bimap`, `ordNub`, `partitionEithers`, `eitherDecodeStrict'`) up to whole subsystems. If a one-line standard call replaces an in-house helper at the call site, inline it and delete the helper. Helpers should earn their keep across many call sites — single-use shims that rename or lightly wrap a library function belong nowhere. Before writing non-trivial logic, check Hackage/Stackage for an existing package; adding a dependency almost always beats hand-rolling and maintaining the equivalent. Fewer lines is a goal in itself — leaning on `base`/`relude`/`aeson`/`bytestring` and the wider ecosystem is how we get there. - **Prefer concrete typed handler return types over opaque `Html ()`.** A handler that returns `RespHeaders MyPageGet` (a newtype carrying `PageCtx` + the data it renders) is testable — tests can inspect the payload structurally, golden-test the rendered HTML, and refactor the renderer independently. A handler that returns `RespHeaders (Html ())` collapses the data and view into one opaque blob you can only assert against as a string. **Do not "simplify" typed-newtype handlers (`ApiGet`, `ManageMembers`, `CreateProject`, …) down to `Html ()` for the sake of LOC.** When introducing new page handlers, give them a typed wrapper carrying the data the renderer needs, with a `ToHtml` instance that calls the render function. Helpers like `withSettingsPage` are fine for handlers that already return `Html ()`, but don't use them as a reason to flatten typed handlers. - **No duplicate functions — extend, don't fork.** When you need a slightly different version of an existing function (e.g. one that returns more fields, takes an extra filter, or aggregates more columns), extend the existing function and update its callers. Do **not** add a parallel `fooWithBar` / `fooV2` / `getXAndY` next to `getX`. Two near-identical SQL queries / handlers / helpers will drift, and a future reader has to diff them to figure out which one to use. Widening a return type or adding an argument is almost always cheaper than maintaining the second copy. - **Extend the shared component; never build a one-off beside it.** The same rule as "no duplicate functions", applied to components (`Widget`, `Table`, `TimePicker`, `emptyState_`, the page shells). When a component *almost* does what a page needs, the answer is to add the capability to the component so every caller gets it — **not** to hand-roll a bespoke HTMX fragment, a private renderer, or a page-local variant next to it. "It only renders that server-side" / "that path doesn't populate it client-side" is a description of a gap to close in the component, not a reason to reinvent it one page over. A capability added to `Widget` is one implementation, tested once and available everywhere; a page-local copy is a second implementation that will drift and that the next reader has to diff. Reach for a new surface only once you have established the shared one genuinely cannot be extended to cover the case, and say why in the commit. - **Relude** as custom Prelude (`NoImplicitPrelude` + `import Relude`) - **GHC2024** language standard (includes `DataKinds`, `DerivingStrategies`, `GADTs`, `LambdaCase`, `RoleAnnotations`, etc.) with additional extensions in `package.yaml` - **Qualified imports** for most non-Prelude modules - `-Weverything -Werror` with selective `-Wno-*` suppressions - `UUIDId` newtype for typed IDs (e.g., `ProjectId`, `UserId`) - **Comments: concise docs that carry information not in the code, or none at all.** A comment earns its lines only by stating what the code cannot — an invariant, a wire contract, why a surprising choice was made. Comments that restate the code, narrate a change, or pad an obvious helper are bloat; delete them rather than polish them. - **Record dot syntax** — always use `foo.bar` (OverloadedRecordDot) instead of accessor functions `bar foo` - **No record field prefixes** — use bare field names (`key`, `since`, `expression`) not prefixed names (`csKey`, `mqSince`, `mcExpression`). `DuplicateRecordFields` is enabled to allow the same field name in multiple records within a module. - **Always derive instances** — never write manual `ToJSON`/`FromJSON`/`ToField`/`FromField` instances when deriving can work. Use `deriving-aeson` (`CustomJSON`, `FieldLabelModifier`, `StripPrefix`, `CamelToSnake`, `OmitNothingFields`) for JSON. Use `WrappedEnumSC` (from `Pkg.DeriveUtils`) for enum-like types needing `ToJSON`/`FromJSON`/`ToField`/`FromField` with snake_case constructor stripping. Prefer `DerivingVia`, `DeriveAnyClass`, and `deriving newtype` over manual instances. - **SVG icons via sprite sheets** — render icons with `faSprite_ "icon-id" "solid" "w-4 h-4"` (or `"regular"`). The icon id must exist as a `` in `static/public/assets/svgs/fa-sprites/solid.svg` or `regular.svg`; referencing a missing id silently renders nothing. We have a **FontAwesome Pro** subscription (kit token in `.env` as `FONTAWESOME_PRO_TOKEN`). To add a new icon: 1. Find the icon name at fontawesome.com (e.g. `arrow-right`, `circle-check`). Check the style: `solid` or `regular`. 2. Run `make fa-add ICON=icon-name STYLE=solid` (or `STYLE=regular`). This appends the `` to the matching sprite file automatically. 3. Use `faSprite_ "icon-name" "solid" "w-4 h-4"` in Haskell. `make fa-add` (→ `scripts/fa-add.py`) fetches from the **FontAwesome Pro** API first (using `FONTAWESOME_PRO_TOKEN`, read from env or `.env`), falling back to the FA 6.7.2 **free** CDN only if the token is absent or the icon isn't in Pro — so Pro-exclusive icon names download with no manual steps. The token is env-only; never hard-code it. Sprite files are cache-busted automatically via `hashAssetFile`. Only `solid` and `regular` sprite files are wired into `faSprite_` (Utils.hs) today. To use a new Pro *style* (`light`/`thin`/`duotone`), `make fa-add` will download it into `