Commit graph

25 commits

Author SHA1 Message Date
Anthony Alaribe
7a13fc0887 chore(scripts): share the lost-work sweep instead of keeping it on one laptop
Written after `git add -A` + `reset --hard` destroyed another session's
uncommitted work on 2026-09-02 and every ref-based check came back clean —
`git log --all`, branch diffs and prod-vs-master HTML can only see work that
reached a ref, and a dangling commit has not.

- `dangling-work.sh` lists commits no ref can reach, and (PRESERVE=1)
  snapshot-tags every dirty worktree via `git stash create`, which builds the
  commit object WITHOUT touching the index or working tree — safe against a
  checkout another session is editing. Its first run tagged 11 dirty checkouts,
  several 8+ days stale, none of them recoverable if anyone had reset them.
- `dangling-triage.sh` answers "is this orphan actually lost?" by the identifiers
  a commit introduced rather than its lines. Line comparison is useless here:
  fourmolu and the deep-clean sweeps reword everything, so landed work scores
  100% missing. Identifiers survive refactors and scored 14 of 18 correctly.

Out of scripts/local/ (gitignored) because neither reads a secret or touches a
database — they are pure git, and the whole point is that someone else can run
them. The repo list is now derived from this checkout and its siblings rather
than hardcoded to one machine's home directory.
2026-09-03 09:52:39 +02:00
Anthony Alaribe
c2860941b2 Put the e2e suite in CI, and fix make ci resolving its root to /
`make ci` has been broken since 70ed1780a. That commit runs ci.sh from a copy at
/tmp so the script can be edited mid-run, but line 29 derives the repo root from
the script's own path — from /tmp that resolves to `/`, so every check died with
"unknown check ... (have: )" before doing any work. cmd_local now passes
CI_ROOT=/build, which the container's working_dir already is.

With that fixed, e2e joins ci/checks.tsv as a real check: it requires ghc, node
and pg, and runs in the same job as integration-tests because that is the only
one with all three. scripts/e2e.sh starts its own server, so the check only has
to supply the binary and chromium. Both workflows gate on it, and the gate lists
it.

Verified by running it in CI's own containers rather than by inspection: 48
passed on linux/aarch64, attested to
refs/ci-attest/v1/e2e/398ebd19.../linux-aarch64/ghc.minio.node.pg.

scripts/e2e.sh now honours DB_HOST/DB_PORT, which is what CI sets and what its
`pg` capability probes.
2026-08-20 01:40:34 +02:00
Anthony Alaribe
fb4027edef Fix empty team picker 400, KQL pipe suggestions, and make e2e reproducible
An empty team picker posts `teams=` — one empty value meaning "none selected" —
which the derived [UUID] decoder read as a malformed UUID and rejected the whole
form over. Creating a dashboard, a widget alert, or an alert without assigning a
team answered with a bare 400 page instead of the toast those handlers raise for
every other kind of bad input.

Utils.FormWithOptional carries the fix as a DerivingVia wrapper, so all three
forms stay derived — field names, Maybe handling and repeated values are still
whatever the generic instance does. Four doctests pin the invariant, including
that other keys are untouched and an absent key is not invented; those run in
CI's doctests check, unlike Playwright.

Also:

- KQL suggestions after `|` offered comparison operators instead of commands.
  The segment-boundary branch only handled the first segment.
- ~50 suggestion tests asserted nothing: they reached for
  monaco.languages.CompletionItemProviderRegistry, an internal that no longer
  exists, behind an unguarded `if`. They now drive computeSuggestions through
  the component's own schemaAccess.
- scripts/e2e.sh runs the suite against a throwaway server on 8081 built from
  .env.example, so it cannot reach the production database that .env points at.
  playwright.config.ts no longer defaults to 8080, which is what `make
  live-reload` serves. 48 specs, green from a dropped database and a warm one.
- New spec puts one widget of every type on a canvas and checks each is adopted
  by GridStack and survives a refresh; dashboard-grid.spec.ts now builds its own
  fixture instead of opening whichever dashboard sorted first.
- Live-tail specs asserted a Start button and service gate the component
  deliberately does not have; billing/onboarding used renamed plan-card ids.
- Four tests for <live-tail>'s follow behaviour: rows arriving must not yank a
  reader who scrolled up.
2026-08-20 00:35:08 +02:00
Anthony Alaribe
70ed1780a8
ci: run local CI from a snapshot, so editing ci.sh mid-run is safe (#497)
bash reads a script incrementally by file offset. The container ran ci.sh
straight off the bind mount, so editing scripts/ci/ci.sh while a run was in
flight made the running shell resume at a stale offset and die with

  scripts/ci/ci.sh: line 605: syntax error near unexpected token `;;'

The check itself had already passed and been attested; only the wrapper
died, which is precisely the "bookkeeping must not fail a green check"
failure this repo has hit twice now. And a `make ci` lasts tens of minutes,
so editing the script meanwhile is a normal thing to do, not a mistake.

Copy it into the container and exec the copy.
2026-08-16 21:19:07 +02:00
Anthony Alaribe
2039c53e1a
ci: one ghc-options string, so test steps stop recompiling the whole build (#495)
`build` compiled with `-O0 +RTS -A64m -n2m -RTS`; doctests, unit-tests and
cli-tests compiled with just `-O0`. cabal's build plan hash includes
ghc-options, so those are two different configurations and every test step
recompiled all ~184 modules that `build` had just compiled. That is most of
the wasted time in the job — and it was inherited from the workflow, not
introduced here.

It is also a memory cliff. Without `-A64m -n2m` GHC uses its default
allocation area, so the redundant recompile is far more GC-heavy, and on a
4-vCPU runner it dies with no error message at all — the log jumps from
"[182 of 184] Compiling …" straight to a failed step. That is exactly what
blocked the deploy of b052d6cf4: `build` passed, then `doctests` rebuilt
everything and was killed. The previous master run passed the same step,
because it had ~4.5k fewer lines to recompile.

One string everywhere. The test steps now reuse what `build` produced
instead of redoing it.
2026-08-16 20:58:28 +02:00
Anthony Alaribe
7ec090c51b
ci: reuse an already-built image, and document running CI locally (#493)
With tests cached the deploy is 4m22s, of which 3m52s is the image build — the
whole critical path. The image is not fingerprinted like the other checks
because it produces an artifact rather than a verdict, but it does not need to
be: it is tagged with the commit SHA, so the registry is already its cache and
the SHA is already its identity.

build-image now probes the registry and skips when the tag exists. Nothing
changes for a new commit; it collapses the repeat cases to ~30 seconds — a
re-run of a failed or cancelled deploy, a workflow_dispatch of an existing
commit, a revert to an already-built SHA. Skipping re-points :latest at the
reused image rather than leaving it on an older commit, since docker-compose.yml
pulls :latest.

The same probe lets a bigger machine pre-build the image (`make deploy-image`,
linux/amd64 — Rosetta on Apple Silicon, not QEMU). That moves the artifact that
runs in production outside CI, and unlike an attestation it cannot be verified
afterwards, because a Haskell build is not bit-reproducible. So the build
refuses a dirty tree, and the deploy job prints `built by: …` every time —
never fatally, since a provenance lookup must not be why a gated build fails to
reach production.

Docs: CONTRIBUTING gains the section a contributor needs, including the two
caveats that would otherwise surprise them — the first `make ci` is a cold
build, and publishing a result needs push access, so from a fork it is fast
local feedback but CI re-runs the checks itself. README links docs/local-ci.md.
2026-08-16 20:24:40 +02:00
Anthony Alaribe
d8242d549b
ci: stop emitting the run id twice in an attestation's provenance (#492)
Every attestation CI has published so far carries

  runner=github-run-3194584899031945848990

which is `github-run-` followed by run id 31945848990 twice, so the one
field that traces a proof back to the run that produced it cannot be
pasted into a URL.

`${X:+github-run-$X}${X:-$(whoami)@$(hostname)}` reads like an if/else and
is not one: the `:-` arm yields $X itself when X is set, so a set
GITHUB_RUN_ID takes BOTH arms. Spell the branch out.
2026-08-16 20:12:05 +02:00
Anthony Alaribe
f8514ee295
ci: let anyone run CI, and let CI skip what was already proven (#491)
CI re-runs work that has already been done: a PR run proves a tree, the squash
merge keeps that exact tree, and master builds and tests all of it again. On
4-vCPU runners that is ~20 minutes per push, and none of that capacity is
available to a developer whose laptop is faster than the runner.

ci/checks.tsv is now the one definition of what CI is; the workflows and
scripts/ci/ci.sh both read it. Each check is fingerprinted over the git blob
hashes of its declared inputs, and a pass publishes an attestation as a ref:

  refs/ci-attest/v1/<check>/<fingerprint>/<platform>/<capabilities>/<date>

A gate job resolves those with one ls-remote and no object fetches, and skips
any check already proven for the tree it is about to test — whether by an
earlier run, a re-run, or `make ci` on someone's machine. Producing one needs
push access, the same trust boundary as pushing code; forks can neither create
nor be affected by them.

Reuse is deliberately conservative because this gates a production deploy.
Checks declare the capabilities they need and attestations record the ones the
environment had, so a run that fell back to a stub service can never satisfy a
check that needs the real one. Fingerprints are pinned from the gate, so steps
that rewrite the tree (hpack, npm ci, hlint --refactor) cannot desync what gets
attested from what was looked up. The deploy condition is spelled out rather
than implied by `needs`: build-and-test may be skipped, but only when the gate
says every Haskell check was already proven.

Also switches the minio service to bitnamilegacy/minio. The bitnami/minio
repository was emptied in the Aug-2025 catalog change and now has no tags at
all; CI has only been working off a runner-side image cache, and an eviction
would have taken the integration suite down with nothing in the diff to blame.

Verified end to end: all six checks ran through ci.sh and published
attestations, and a clean checkout on macOS/arm64 then resolved CI's
linux-amd64 attestations and skipped the entire Haskell suite.

See docs/local-ci.md.
2026-08-16 19:20:59 +02:00
Anthony Alaribe
7c01233ed6 Slim the CLI binary and rebuild its release pipeline around it
The CLI no longer needs the frontend build, proto codegen, or the eight
Homebrew packages the macOS job used to install — none of that is reachable
from monoscope-cli. Removing openapi3 from the shared package (a CLI has no use
for schema generation; the ToSchema instances move to Pkg.DeriveUtils and the
new Web.WireSchemas) takes the stripped -O2 binary from 107MB to 97MB.

- cli-release.yml: builds monoscope-cli:exe:monoscope only, with -O2
  -split-sections plus --gc-sections/-dead_strip and strip.
- scripts/check-cli-linkage.sh fails the build if the binary links libpq,
  librdkafka, grpc, protobuf or a compression library — the no-prerequisites
  promise is now enforced rather than assumed. It also rejects /opt/homebrew
  paths, and the macOS job repoints Homebrew's zlib at /usr/lib and re-signs,
  which would otherwise have shipped a binary that only runs on a Mac with brew.
- cli-tests now run in the main CI workflow; make cli-build/cli-test/
  cli-release-build for local use.

Measured but not taken: dropping brick/vty saves only 4MB, which does not pay
for losing the arrow-key project picker at login.
2026-08-06 01:49:06 +02:00
Anthony Alaribe
49786a737a chore(dlq-redrive): log fetch progress, add KGO_DEBUG franz-go logger
Set KGO_DEBUG to surface franz-go's own debug output when a re-drive stalls
and it isn't clear whether the client is fetching at all.
2026-07-29 21:32:34 +02:00
Anthony Alaribe
4570dfa709 ops(dlq-redrive): raise BrokerMaxWriteBytes to 160 MiB
franz-go rejects producer init when BrokerMaxWriteBytes (default 100 MiB) is
below ProducerBatchMaxBytes (raised to 120 MiB for 100 MiB replay records).
2026-07-10 12:42:10 -07:00
Anthony Alaribe
e270ff03dc feat(replay): support up to 100 MiB session-replay messages
Raise the replay message-size ceiling end-to-end so full rrweb session
recordings land instead of being parked as replay:oversize_message poison
(2026-07-10: the 10 MiB cap was silently dropping a steady drip of 10-41 MB
rrweb blobs into otlp_deadletter-parking).

- Replay.hs: maxReplayMessageBytes 10 MiB -> 100 MiB (payload accept cap)
- Queue.hs producer: message.max.bytes 64 -> 120 MiB, receive.message.max.bytes
  100 -> 160 MiB (socket buffer must exceed message cap; 20 MiB header headroom)
- Queue.hs consumer: max.partition.fetch.bytes / fetch.max.bytes 64 -> 120 MiB,
  receive.message.max.bytes -> 160 MiB
- BackgroundJobs.hs parking-redrive consumer: same 120/160 MiB caps
- dlq-redrive tool: BrokerMaxReadBytes 32 -> 160 MiB (the hard socket cap that
  must clear the largest message; KIP-74 returns oversized singles beyond the
  small WAN-friendly fetch limits), ProducerBatchMaxBytes -> 120 MiB

Broker side raised in tandem (rpk): cluster kafka_batch_max_bytes and the
rrweb-client + otlp_deadletter{,-parking,-retry-60s,-retry-600s} topic
max.message.bytes overrides all set to 120 MiB.
2026-07-10 09:51:47 -07:00
Anthony Alaribe
aea4921987 ops(dlq-redrive): mainline-source backfill mode + src==dst loop guard
Additions from the 2026-07-09 a0c2640 incident backfill (TF acked inserts
with no DLQ copy; TS + mainline Kafka were the only sources):

- -synth-tf-headers: consume a raw OTLP mainline topic (no headers) and
  synthesize the DLQ-replay contract (ce-type via topicToCeType mirror,
  original-topic, monoscope-write-failure=tf-failed, pg-succeeded=true) so
  the carousel retries ONLY the TimeFusion leg. Guarded to mainline topics.
- HARD offset bound at the snapshotted window end inside EachRecord: kgo
  keeps fetching past the window, and with src==dst the consumer reaches
  its own re-produced copies — remaining-count alone let it self-amplify
  (555k excess produced before the kill).
- Used to backfill 02:42-03:26 UTC: 103,395 otlp_spans + 56,322 otlp_logs
  re-driven, zero failures.
2026-07-09 04:31:53 -07:00
Anthony Alaribe
35649daa01 ops(dlq-redrive): track the re-drive tool; timestamp-range mode, stall watchdog, resumable state
The tool the DLQ runbook's recoveries B/B2 depend on lived only in the
gitignored scripts/local scratch dir — force-added so the documented
procedures are runnable from a fresh checkout (binary stays ignored).

Changes from the 2026-07-09 TF backfill run (4.77M messages re-driven):
- -from-ts/-to-ts: re-drive a time slice of the base DLQ itself; window
  end is hard-bounded per record — with src==dst, counting alone lets the
  consumer reach its own re-produced copies (self-amplifying loop)
- -state: checkpoint the ACKED per-partition frontier (flush barrier)
  every minute; restarts resume instead of re-producing the window
- 3-min produce-stall watchdog (WAN runs wedge ~10-15min in; exit for a
  wrapper retry loop) and 8MB fetch sizes (64-128MB wedges over WAN)
- -trim refused outside whole-window parking re-drives
2026-07-08 19:25:42 -07:00
Anthony Alaribe
f0fc0c0351 fa-add: fetch FontAwesome Pro API first, free CDN fallback 2026-07-04 09:57:07 -07:00
Anthony Alaribe
ed1c5dfc7a Log Explorer: CSS-driven JSON-tree collapse, field-menu cleanup, fa-add tooling
- LogItem: rewrite request/service/error extraction via aeson-lens; typed
  detail/http/json tab components + empty states
- Utils: single collapse/expand toggle button (bulk-writes .tree-toggle checked
  state); reorder field context-menu actions, add "Copy key == value"
- Parser: skip user columns that resolve to a default SELECT expr (dedup); allow
  normalizeKeyPath alias chars in splitTrailingAlias
- session-replay: compact header, icon-only utility buttons with tooltips
- Add `make fa-add` + scripts/fa-add.py to append FA symbols to the sprite sheet
2026-07-04 03:29:42 -07:00
Anthony Alaribe
d84fbff1a8
fix(charts): don't advance query-cache watermark on failed delta fetch (#456)
* fix(charts): don't advance query-cache watermark on failed delta fetch

Time-series chart widgets could render permanent holes: in the partial-hit
path, a delta fetch that failed (timeout / TF planning error / dropped conn,
swallowed into an empty MetricsData by withChartSpan) still advanced cached_to
to the request end. That orphaned the [cachedTo, reqTo] window forever, since
every later delta starts from the advanced watermark — so short ranges rendered
near-empty while wider ranges (a separate bin_interval cache entry) looked fine.

Gate updateCache on `isNothing deltaData.error` in both the PartialHit and
CacheMiss branches, so the watermark never advances past data we actually
fetched. On a transient failure the next poll re-fetches [cachedTo, now] and
backfills the gap.

Regression tests in QueryCacheSpec: failed delta doesn't advance cached_to and
still serves the cached rows; a successful-but-empty delta still advances.

* minor updates
2026-07-02 09:53:51 -07:00
Anthony Alaribe
d0350afa52
refactor: CLI agent-mode hardening + cross-cutting LOC reduction (#365)
* refactor: extract mkPageCtx page-handler bootstrap

Replaces 25+ inlined BWConfig + sessionAndProject + ask AuthContext
sequences across Pages/* with a single mkPageCtx helper that returns
(Session, Project, BWConfig) seeded with the always-set fields
(sessM, currProject, config). Net: -102 LoC.

* refactor: extract toastError helper for validation/error responses

Collapses the recurrent 'addErrorToast msg Nothing >> addRespHeaders payload'
shape across Pages/Projects.hs and Pages/Dashboards.hs. Net: -5 LoC plus
+5 LoC for the helper and re-export = breakeven, but eliminates a duplicated
two-line pattern at every site.

* refactor: extract withCardFrame for widget renderer card shell

Collapses the duplicated outer flex-wrapper + optional widget header +
flex-1 fill + surface-raised card shell shared between renderLogsWidget,
renderTraceTable, and renderTable into a single helper.

* refactor: derive FieldTypes/FieldCategoryEnum JSON via WrappedEnumSC

Replaces manual ToJSON/FromJSON instances and the parseFieldTypes,
parseFieldCategoryEnum, fieldTypeToText helper functions with a single
deriving-via clause that piggybacks on the existing WrappedEnumSC wrapper.
Net: -50 LoC.

* refactor: distill CLI; reuse server types (Schema, Paged, MetricsData)

Replace ad-hoc AE.Value field-name probing in the CLI with typed parsing
against the server's existing response types so wire-shape changes
surface as compile errors:

- filterSchema now takes Models.Telemetry.Schema.Schema
- normalizeList decodes via Web.ApiTypes.Paged
- metrics renderers/assertion take Pages.Charts.Types.MetricsData

Also: factor validateDurationOrDie/validateAndNormalizeKind helpers,
fold three case-of-OutputMode triplets through renderWith, derive
ServiceRow/ConfigGetJson/AuthStatusJson/TraceSummary JSON via CustomJSON.

* Auto-format code with fourmolu

* docs: add agentic-pipeline showcase to README

Show the discover -> search -> context -> triage chain with the stable
JSON envelope each command exposes, so a reader scanning the README sees
why the CLI is agent-friendly before getting to the Claude Code Skills
section.

* refactor(cli): apply hlint fixes (errors + warnings + suggestions)

Drops 2 errors and 7 warnings from `hlint cli` to 0:
- Resource.hs: `>> pure n` -> `$> n` for the apply-batch counter
- Commands.hs: `ExceptT . pure` -> `hoistEither`; manual `maybe ... Right` -> `maybeToRight`
- Config.hs: `either (const x) id` -> `fromRight x`
- UI.hs: `T.unlines` -> Relude's `unlines`
- Validate.hs: drop redundant `void`

Plus higher-impact suggestions:
- `data` -> `newtype` for ServicesCommand, ConfigGetOpts, ServicesListOpts, FormEditorName
- `fmap (fmap f) x` -> `f <<$>> x` in 6 list/schema/facets handlers
- `Set.singleton` -> Relude's `one`
- `pure ()` -> `pass` in UI event handlers
- `\_ -> action` -> `const action`
- `(\c -> Map.lookup c idxMap)` -> `(`Map.lookup` idxMap)`

Build green: `All good (8 modules)`. The two remaining hlint hints
(`Data.Set as Set` alias, `universe` import) belong in `.hlint.yaml`
rather than the code.

* fix(test): unwrap IssueL through .base.id in ErrorPatternsSpec

IssueL is a list-view wrapper around Issue (`{ base :: Issue, ... }`),
so the `id` field is reached through `.base`. Four sites in
ErrorPatternsSpec called `issue.id` directly, which type-errored after
IssueL gained its current shape. Matches the pattern used in
src/Pages/Anomalies.hs (`errL.base.id`).

* fix(cli): accept uppercase --since suffixes (1H, 24H)

The platform's TimePicker emits uppercase ("1H", "24H") — the same
strings the CLI itself injects as defaults via buildSearchParams /
runServicesList — but the validator only accepted lowercase, so a user
who copy-pasted the displayed default got rejected client-side. Fold
the suffix to lowercase before checking the whitelist; preserve
original casing in the returned value.

Add a regression test in CLIE2ESpec asserting `--since 1H` is accepted.

* fix(cli): four review nits — flag-aware duration error, single decode, doc/comment fixes

1. validateDurationOrDie now takes the flag name, so --window failures
   read "--window must match …" instead of "--since must match …".
   Three call sites updated (--since x2, --window x1).

2. aggregateServices keeps its `Map.lookup "service"` but with a comment
   pinning the contract: the server aliases resource___service___name to
   the short "service" column in colIdxMap (Pages.LogExplorer.Log.allCols).
   Added an e2e regression in CLIE2ESpec that asserts services-list still
   returns {services, count} so a column rename can't silently zero the
   output.

3. docs/cli.md agentic-incident-workflow: replace .items[] with .data[]
   in the issues-list and log-patterns-list jq pipes — they'd break
   against the now-normalised list envelope.

4. runEventsContext --summary used to walk logsData twice (once in
   normalizeEventsResponse, once in withTraceSummary). Introduce a
   `DecodedEvents` record that carries idxMap/rows/raw, decode once at
   the call site, and thread it through both functions.

* perf(cli): back services list with /api/v1/facets instead of 10k-event scan

Previously runServicesList fetched up to 10,000 events and aggregated
client-side because /api/v1/metrics returned empty for `summarize ... by`.
The facets endpoint (precomputed in apis.facet_summaries via a background
job) is one indexed lookup — sub-millisecond regardless of project size.

- Drop the events-scan fallback; switch to GET /api/v1/facets?field=resource.service.name
- ServiceRow loses the `last_seen` field (facets don't track per-value
  recency). The remaining {name, events} is what the use-case actually
  needs ("what services exist? how busy are they?").
- e2e regression in CLIE2ESpec already covers the {services, count}
  envelope; comment updated to pin the facets-backed contract.
- docs/cli.md and monoscope-skills/README.md output-shape tables drop
  last_seen.

* chore: trailing review nits — unused asks, V.take, normaliseList warning, debug-comment

Server side:
- dashboardsGetH: drop the appCtx ask; bw.config.liveReloadDashboards
  carries the flag now that mkPageCtx threads config through.
- teamAlertsGetH: drop appCtx + sess + project, all genuinely unused
  bindings.

(dashboardGetH / dashboardTabGetH / dashboardTabContentGetH /
unifiedMonitorOverviewH still bind appCtx — they read appCtx.env.* which
isn't on bw.config, so the binding is required.)

CLI:
- takeFirstEvent: V.take 1 arr instead of take 1 (toList arr); avoids the
  Vector→List→Vector round-trip.
- Resource.normalizeList split into normalizeListE returning Either, with
  Left carrying the raw payload on shape mismatch. runList now flags the
  unrecognised case via printDebug so contributors see when the server
  envelope drifts.
- Core.renderParams: comment pinning the "debug-only, NOT URL-encoded,
  redact if you ever pass auth via query params" contract.

* fix(cli): CI doc uses .data, emitFirstEventId avoids list round-trip

- docs/cli.md CI/Automation example said `jq '.items | length'` —
  inconsistent with the agentic-workflow section above and broken since
  normalizeList moved everything to `.data`.
- emitFirstEventId now uses `arr V.!? 0` instead of `head (toList arr)`
  — same V.take optimisation as takeFirstEvent, no intermediate list
  allocation.

Bug #2 from the review (renderEventsTable in runEventsContext receiving
raw `val`) is intentional: the table renderer reads `cols`/`logsData`
out of the raw envelope to produce a dense table; the JSON path uses
the named-field `enriched` form. Passing `enriched` to the table
renderer would render an empty table because the normalised shape
doesn't carry `cols`/`logsData`. The split is by design — table mode is
column-oriented, JSON mode is record-oriented.

* fix(cli): six review items — agent-mode truthy check, --first table parity, single decode, derived facets, forall pinning

3. isAgentMode no longer fires on `CI=` (empty) or `CI=false` — many dev
   shells export CI unconditionally; treat empty/"false"/"False"/"FALSE"/"0"
   as off.
4. --first parity in table mode: takeFirstRow trims the raw envelope's
   logsData to one row, so table output matches the JSON path's
   takeFirstEvent slice instead of silently dumping all rows.
5. Free-floating Haddock above DecodedEvents merged into the actual
   docstring (the orphan would have triggered a -Wmissing-signatures-ish
   warning eventually).
6. runEventsSearch now decodes once and threads DecodedEvents through
   normalizeDecoded, matching the pattern in runEventsContext. No more
   double walk of logsData on the JSON path.
7. parseServiceFacets decodes via Models.Apis.Fields.FacetValue
   (FromJSON via DAE.CustomJSON) — drops 6 lines of hand-rolled
   KM.lookup/AE.String/AE.Number unpacking and surfaces server wire
   changes at compile time.
8. withMetricsData now forwards aeson's Error message through
   printDebug so a server envelope drift is visible under --debug
   instead of vanishing into the raw-JSON fallback.

Plus: explicit `forall es a b` on apiGetJson/apiPostJson/apiPutJson/
apiPatchJson so TypeApplications at call sites
(`apiPostJson @_ @_ @AE.Value`) survive future constraint reordering by
fourmolu/hlint. Without this, alphabetical resort of
`(AE.FromJSON b, AE.ToJSON a, Environment :> es, ...)` flips the
implicit type-var order and the third `@AE.Value` lands on `es` instead
of the body type.

Run fourmolu over the touched files; #10 (double blank lines) is
auto-fixed.

* chore: round-three review nits — null instead of "", helpCases coverage, dead code

3. normalizeDecoded now emits `null` for missing fields (row shorter
   than colIdxMap) instead of silently coercing to "". Agents can now
   tell "field present but empty" from "field missing".
4. Comment on the --debug setEnv site explaining the single-shot, pre-
   thread-spawn placement (so future readers know it's not a race).
6. emitCompletion lifts to IO once at the entry instead of per-branch.
7. CLIBinarySpec helpCases extended with `facets`, `schema`, `services
   list`, `events {search,context,tail}`, `metrics {query,chart}`,
   `auth login`, `completion` — every top-level subcommand now has a
   --help smoke test.
8. dashboardGetH/dashboardTabGetH: replace appCtx.config.hostUrl with
   bw.config.hostUrl now that mkPageCtx threads config through; appCtx
   binding stays because appCtx.env.apiKeyEncryptionSecretKey isn't on
   bw. dashboardTabContentGetH untouched — no mkPageCtx, no bw.

Style:
- Drop dead extractNumericRows from Pkg.CLIFormat (no callers since the
  metrics path moved to MetricsData).
- Export validateDurationFor so the doctest examples (which reference it)
  resolve at doctest runtime.

Re #2 (parseServiceFacets via FromJSON) and #10 (double blank lines):
already landed in commit 2a54bd0b.

Re #5 (renderParams allowlist): keeping the comment-only enforcement.
The CLI doesn't put credentials in query params today (Bearer + project
go via headers), and a type-level distinction would touch every API
helper for a hypothetical leak. The doc note is sufficient guard for
contributor review.

* fix(cli): trace-summary error count, KQL injection guard, small wins

Core bug:
- withTraceSummary used `cell "severity"` to count errors per trace, but
  the events colIdxMap (Pages.LogExplorer.Log.allCols) doesn't expose a
  "severity" column — only the boolean "errors" flag. Result: every
  trace summary reported error_count=0 regardless of traffic. Now reads
  cell "errors" and treats "true" as 1. Also added a defensive fallback
  for service ("service" <|> "resource.service.name") in case a future
  column-set rename hides the alias.

KQL safety:
- runEventsGet escapes `\` and `"` in opts.eventId before splicing into
  the query string. Event IDs aren't UUIDs (synthetic-… span ids,
  log_pattern integers, …) so format validation is wrong; escaping
  defends against injection regardless of shape.

Small wins:
- isAgentMode: fold suffix to lowercase once via `map toLower v` rather
  than enumerating "false"/"False"/"FALSE".
- filterSchema: `Map.take n filtered` instead of
  `Map.fromList (take n (Map.toList filtered))` (containers ≥ 0.6.3).

Doctests note:
- The earlier "doctest comment" the review flagged was already removed
  in a prior commit. Pure validators are still covered end-to-end via
  CLIE2ESpec ("--since 1xyz", "--kind banana") which exercise the real
  binary. Wiring CLI/* into the lib doctest target would need moving
  the modules into src/ — out of scope for this round.

Pushed back:
- AuthStatus method ADT: the strings "env"/"token" appear in two
  adjacent lines of one function. A two-constructor ADT + custom ToJSON
  is more boilerplate than the leak it prevents (no other consumer).
- Map merge tuple readability: a record would cost two lines and one
  data declaration to save zero compile errors. The merge fits on one
  line; leaving it.

* chore: clarify intent — `mempty` empty bodies, capFacets exhaustiveness comment

- Pages/Projects.hs: replace seven `toastError "…" ""` and three
  `addRespHeaders ""` with `mempty`. Reads as "no body" rather than
  "empty string body" — the underlying type is `Html ()` whose Monoid
  identity is the empty Html, same wire effect.
- cli/Main.hs `capFacets`: comment on the trailing wildcard clause so
  the (Just n) + non-Object fallthrough is explicitly documented vs the
  Nothing case the first clause already handles.

Re facets in CLIBinarySpec helpCases: already added in 76ebf71c.

* chore: clear hlint warnings across src/

BackgroundJobs.hs: 6× displayException + 2× fromException were
resolving to Control.Exception.* (because the file did
`import Control.Exception (ErrorCall (..))` which exposed the
Control.Exception namespace to GHC's resolver and tripped
.hlint.yaml's "use Relude version" rule). Drop the unqualified
import; ErrorCall now goes through `Control.Exception qualified as CE`
so the only `Control.Exception` reference is the explicit qualifier
and unqualified displayException/fromException resolve through Relude.

Pages.Anomalies.hs:
- NE.head -> head (Relude)
- two whenJust lambdas -> point-free (toHtml composition)
- when (... `notElem` ...) -> unless (... `elem` ...)

Pages.Settings.hs: sortOn fst -> sortWith fst (no caching needed for
projection on first component).

Pages.Telemetry.hs: MapS.singleton k v -> one (k, v) (Relude polymorphic
container constructor).

Pages.LogExplorer.Log.hs: two `\j -> (j, x)` lambdas -> tuple sections
(TupleSections is part of GHC2024, already on).

Verified clean: `hlint src/BackgroundJobs.hs src/Pages/Anomalies.hs
src/Pages/Settings.hs src/Pages/Telemetry.hs src/Pages/LogExplorer/Log.hs`
reports "No hints".

* Auto-format code with fourmolu

* fix(cli): forward Paged decode error, lenient body decode, unambiguous null

3. normalizeListE now returns Either (rawValue, reason) — when an Object
   fails @Paged@ decoding the aeson Error message is captured and
   surfaced through runList's --debug log. Contributors see *what*
   shape changed instead of a generic "did not match envelope".
   normalizeList stays pure with `either fst id . normalizeListE`.

4. httpExToError now uses TE.decodeUtf8With lenientDecode for both the
   status reason and the response body. The body field of
   StatusCodeException is the truncated raw bytes (≤1024B) and may be
   gzip leftovers, HTML, or arbitrary upstream junk; strict decodeUtf8
   would throw at the error-handling boundary, masking the real HTTP
   failure with a UnicodeException.

6. isAgentMode: replace `not (null v)` with `v /= ""` to dodge the
   Prelude.null vs Relude container-null ambiguity. v is String here.

Re prior style notes:
- Double blank lines between top-level bindings: produced by fourmolu
  format pass; project convention is enforced by the formatter.
- AuthStatusJson / ConfigGetJson: kept as records — the CamelToSnake
  derivation pins the wire shape (an inline AE.object [...] silently
  drifts when fields are added).
- runConfigGet OutputMode asymmetry vs init/set: noted; init takes no
  output path (interactive form) and set has no payload to format.
  Follow-up if it surfaces in user feedback.
- normalizeList = either fst id . normalizeListE (was either id id;
  shape changed when Left started carrying the reason).
- CLI/UI.hs `currentValue = T.strip . unlines . BE.getEditContents`:
  verified — `unlines` here is the Relude re-export, which is polymorphic
  over the IsList/Foldable instance. Compiles clean.

* fix(test): unwrap IssueL through .base.issueType in AnomaliesSpec

Same pattern as the ErrorPatternsSpec fix in f0ab87d7: IssueL wraps
Issue in `base`, so the issueType field lives at `c.base.issueType`,
not `c.issueType`.

* fix(test): add http-types to integration-tests build-depends

CLIE2ESpec imports Network.HTTP.Types.Status (statusCode) for the
reachability probe but http-types wasn't on the test target's deps.
The library has it (line 173), the test suite didn't inherit it.

* fix(test): drop System.Environment lookupEnv import — Relude already exports it

* fix(cli/test): try @SomeException → tryAny, plus seven review nits

Build break: UnliftIO.Exception.try's forall is (m, e, a); the @SomeException
TypeApplication landed on m and tripped a kind error. Switch to tryAny
(MonadUnliftIO m => m a -> m (Either SomeException a)) — same semantics,
no type-arg juggling. Drops the redundant Data.List (isInfixOf) import
along the way (Relude already has it).

Review items:
- AuthStatus: extract printStatus helper for the env/token branches —
  3 lines repeated → 1 helper.
- apiFacets: replace addUTCTime (-86400) with negate nominalDay (Data.Time
  re-export). No more 86400-second magic.
- capFacets: drop the now-redundant `Nothing` clause; the wildcard at the
  end already covers it (and `Just n` + non-Object).
- parseDurationMs: T.toLower the input via a view pattern so --watch 5M
  is treated like --watch 5m, matching validateDurationFor's case-insensitive
  suffix check.
- reqOpts: collapse the broken multi-line `& W.header "Accept" .~ [...]`
  formatter artifact into a single parenthesised line so subsequent `&`
  chain elements align under W.defaults.
- Resource.runList: route the unrecognised-envelope reason through
  printError (always-on stderr) instead of printDebug — agents
  expecting {data, pagination} need to see when the server returns
  something else.
- DecodedEvents: decode count / has_more / cursor at construction
  instead of carrying the raw KeyMap. Three lookups eliminated, intent
  is local to decodeEvents.

Re minor nits noted:
- isAgentMode/isDebugMode re-read env on each call: kept; called at
  most a handful of times per invocation, no hot path.
- Data.Char (toLower): kept the explicit import — Relude does re-export
  it but the explicit form documents which `toLower` we mean (vs
  Data.Text.toLower which operates on Text).

* fix(test): restore Data.List (isInfixOf) — Relude does not re-export it

* fix(test+ci): two CLI test failures, default e2e suite to the public demo project

CI failures:

1. CLIBinarySpec "auth status exits and shows status message" — when CI
   is set, isAgentMode auto-enables and `auth status` emits the JSON
   envelope ({authenticated, method, api_url, project}). The test
   predicate only matched plain-text "Not authenticated" / "Authenticated".
   Extend it to also accept the JSON shape ("authenticated" key).

2. CLIBinarySpec "completion --help" exited 1. The completion subparser
   declared a positional SHELL argument but didn't attach `<**> helper`,
   so `--help` got consumed as the SHELL value and fell through to the
   "unknown shell" branch. Adding helper makes optparse intercept --help
   and emit usage with exit 0.

Default e2e suite to prod demo project:

CLIE2ESpec.getE2EConfig now defaults @MONOSCOPE_API_URL@ to
https://api.monoscope.tech and @MONOSCOPE_PROJECT@ to the public demo
00000000-0000-0000-0000-000000000000. Only @MONOSCOPE_API_KEY@ has no
safe default (read keys for other projects would leak). The pendingWith
message names the missing variable specifically so it's obvious how to
unlock the suite.

Wired up in CI: pullrequest.yml's integration-tests step now reads
MONOSCOPE_API_KEY from secrets.MONOSCOPE_DEMO_API_KEY. Repo owner needs
to mint a read-only key on the demo project and store it in repo
secrets — then every PR/push runs the suite against prod. Forks (no
secret) still get the CLIBinarySpec smoke tests.

docs/cli.md updated with the new default-driven workflow.

* fix(test): assert specific merge edge instead of fragile global count

ErrorPatternsSpec test 12 ("Full LLM pipeline") asserted
`length mergedTypeErrors == 1` after merging one (childId, canonId)
pair. The query counted ALL TypeError patterns with canonical_id set
across the project, so any earlier test that merged a TypeError made
the count drift up. CI now sees 2 because some upstream change in the
LLM/notify pipeline produced an additional pre-merged pattern by the
time test 12 runs.

Switched to a behaviour-level assertion: query the specific childId's
canonical_id and require it equals canonId. This tests what
`assignErrorsToCanonical` actually does, with no dependence on the
global state earlier tests left behind.

* refactor(cli): lens-based shape ops, guards, precomputed indices

- takeFirstEvent / takeFirstRow rewritten with Data.Aeson.Lens
  (`AL.key … . AL._Array %~ V.take 1`). 14 lines of pattern-match-and-
  reinsert collapsed to two one-liners. AL is qualified to avoid
  collision with the `key` record fields on ConfigSet/GetOpts.
- foldFiltersIntoQuery: case (T.null …, T.null …) → pattern guards.
  Reads top-down instead of as a tuple-match puzzle.
- normalizeListE: AEKM.member → has (AL.key …) lens accessor; the
  branch matches on the value directly, no need to destructure
  AE.Object obj first.
- extractColIdxMap: round → floor on the Scientific. Column indices are
  already-integral non-negative ints; floor documents that intent and
  drops banker's rounding semantics that don't apply here.
- parseDurationMs: drop the bare-number-as-seconds fallback. validateDurationFor
  rejects unsuffixed values upstream, so the only paths that reach
  parseDurationMs have a recognised suffix; a typo now returns the
  5s default loudly instead of silently treating "5" as five seconds.
- runMetricsChart: validateDurationOrDie "--watch" before entering the
  forever loop, so a bad --watch string fails at flag parse rather
  than burning HTTP requests every 5s.
- withTraceSummary: precompute svcIdx / errIdx / tidIdx once instead
  of re-Map.lookup'ing the same column path per row. Saves one Map
  lookup per cell per row — measurable on large
  `events context --summary` payloads.

Build deps: cli exe gains `lens-aeson` (already in lib deps for the
server's Data.Aeson.Lens usage). hpack regenerated monoscope.cabal.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-05-02 23:38:22 +02:00
Anthony Alaribe
a3a24fe82e fix: move tmpdir to global scope to avoid unbound variable in EXIT trap 2026-05-02 00:28:21 +02:00
Anthony Alaribe
c13e1b05fc docs(cli): comprehensive CLI docs and monoscope-skills plugin
- Rewrite docs/cli.md to cover all commands: events/logs/traces,
  metrics, services, config, schema, version, global flags, env vars,
  config files, and all command flags previously missing
- Add CLI install quickstart and reference link to README.md
- Add CLI install section to docs/getting-started.md
- Fix scripts/install.sh binary/archive name to monoscope (was mono)

Add monoscope-skills Claude Code plugin (monoscope-tech/skills):
- .claude-plugin/marketplace.json and plugin.json for marketplace install
- skills/kql-reference/SKILL.md — full KQL dialect reference sourced
  from src/Pkg/AI.hs and src/Models/Telemetry/Schema.hs: operators,
  aggregations, time binning, scalar functions, schema fields, examples
- skills/investigate/SKILL.md — incident investigation workflow
- skills/triage/SKILL.md — on-call triage workflow
- AGENTS.md and README.md with install instructions for both
  `claude plugin` and `npx skills add monoscope-tech/skills`
2026-04-30 22:12:59 +02:00
Anthony Alaribe
f257f00355 ci: tie CLI binary releases to regular v* tags 2026-04-30 20:50:12 +02:00
Anthony Alaribe
9cca14c47a ci: add cross-platform CLI release workflow with install script
Builds monoscope CLI binary for linux-x86_64, linux-arm64, darwin-arm64,
darwin-x86_64 on cli-v* tags and publishes tarballs + checksums to GitHub
Releases. Install script updated to match monoscope binary and archive naming.
2026-04-30 20:47:42 +02:00
Anthony Alaribe
f6ba01c8a8
Collapse log-pattern clusters in Inbox and auto-archive stale issues (#361)
* feat(issues): tighten log-pattern signal and add Inbox grouping

Widen the rate-change detector's log-level gate to a case-insensitive
ERROR/WARN/FATAL/CRITICAL whitelist, auto-archive stale log-pattern issues
after 14d of inactivity, and collapse large (type, service) clusters in the
Inbox into expandable group rows. Ship a one-off script that archives legacy
noise (pre-0075 titles, auto-demoted silent drops, stale log_pattern
zombies) without deleting history.

https://claude.ai/code/session_012mpbQgAFGEXNSyGVsmAghh

* Auto-format code with fourmolu

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-04-24 12:30:22 +02:00
Anthony Alaribe
764531ab57 consolidation 2026-03-17 15:24:33 +01:00
Anthony Alaribe
aaeeec505f
Add ANSI-colored summary column rendering for CLI logs search (#340)
* Add ANSI-colored summary column rendering for CLI logs search

Parse field;style⇒value items in summary/latency_breakdown columns
and render them with terminal colors instead of raw JSON arrays.

* Add CLI binary, device auth flow, npm packaging, and CI release

* Auto-format code with fourmolu

* Potential fix for code scanning alert no. 36: Workflow does not contain permissions

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Fix hlint errors and warnings across codebase

Redundant brackets, redundant do in when/unless, sortBy comparing → sortOn,
T.pack $ show → show, when (not x) → unless x, nested if/else → guards,
T.singleton → one, T.unwords → unwords, lambda → point-free style.

* Auto-format code with fourmolu

* Improve CLI: typed config keys, XDG config dir, better parsing and polling

- Replace string config keys with ConfigKey ADT for type safety
- Use XDG_CONFIG_HOME for config directory instead of hardcoded ~/.config
- Improve duration parsing to support ms/s/m/h suffixes
- Use spawnProcess instead of callCommand for browser opening
- Improve sparkline/table rendering and error messages
- Refactor CLI tests for conciseness

* Auto-format code with fourmolu

* fix deps again

* fix deps again

* Auto-format code with fourmolu

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-03-15 02:48:58 +01:00