Commit graph

55 commits

Author SHA1 Message Date
Anthony Alaribe
3027245589
Deploys wait for the Mac's image; image-follow; reliable sign-off; typed dashboard edit refusals (#859)
* Deploys wait for the Mac's image; image-follow builds master's head; CI image bakes ffmpeg/graphviz; dashboard edits refuse missing dashboards, tabs and widgets

- release.sh: the deploy's image step waits up to wait_minutes for the image, and stops early once master moves on
- make image-follow: builds only master's newest head, cancelling a superseded build
- Dockerfile.deps bakes ffmpeg/graphviz; ci.sh's fallback install is bounded
- query-preview test polls for CodeMirror's time-sliced highlighting
- EditRefusal: DashboardGone/TabGone/WidgetGone join EditReadOnly in the route types
- CLI: disambiguate EventsSearchOpts record updates

* log-list test: poll for the load-newer reveal scroll, which lands on a later frame under load

* image-follow: an image already built for the merged tree counts

* hpack: list migration 0278 in monoscope.cabal
2026-11-11 14:56:55 +01:00
Anthony Alaribe
ab21edae74
Workspaces: one build per base, restore from object storage on a miss, build outcomes (#818)
* Workspaces: one build per base, restore from object storage on a miss, builds report how the checkpoint arrived

* Workspaces: restore-then-build reads as one case

* Workspaces: number the run base migration after master's 0275
2026-11-10 21:07:21 +01:00
Anthony Alaribe
77e31fd6a4
Typed follow-ups: git errors, /api/v1 refusals in the route types, typed GET fragments (#802)
* Typed follow-ups: GitConnError, /api/v1 problem envelope declared per route, typed GET fragments

Pkg.Git local validation returns GitConnError instead of Text. /api/v1 refusals and
Servant's own rejections answer one {"error": ApiProblem} envelope, declared per
route via Refuses for OpenAPI/MCP; the CLI prints its message. Remaining
RespHeaders (Html ()) fragments (Telemetry, Issues, CodeContext, Dashboards,
Settings, LogExplorer) return typed view values.

* Review fixes: closed ProblemCode shared with the CLI, split git origin errors, MCP shows refusal messages

ApiProblem moves to monoscope-shared with a closed ProblemCode enum; ApiError
owns every code. Servant rejections no longer guess a field from message text.
Refuses takes one status per combinator. Origin/endpoint/connection errors are
separate types; snippets report ConnectionInvalid. Metric fragments share one
record; the code-mappings picker route returns the fragment it swaps.
2026-11-06 16:58:47 +01:00
Anthony Alaribe
8f7b3438d0 Consolidate src/Pkg (45 -> 29 modules); nested monitor YAML config
Pkg consolidation:
- Slack OAuth, rate limiting and MCP reads -> Pkg.Slack
- ErrorMetrics -> Metrics; Sampling -> IngestBudget; Pkg.UUID inlined into Telemetry
- Rum -> Parser; Facets -> Components.Table; TraceSessionCache -> QueryCache
- PatternMerge -> Drain (stripCodeBlock moves to Data.Effectful.LLM to drop the Pkg.AI edge)
- SchemaLearning Catalog+Hot -> Pkg.SchemaLearning; the flush worker -> Models.Apis.SchemaCatalog
- AgentOperation -> AgentCatalog (AgentAwaitingReview -> CodeReview.Types to break the cycle)
- TestClock -> Data.Effectful.Hasql, beside runHasqlPool
- Icons.TH -> DeriveUtils, icon table -> Utils
- StackTrace folded into ErrorFingerprint: one StackFrame type and one renderer;
  the runtime grammar stays frozen (fingerprints), framesFor is the lossless view grammar
- OperationCall and AgentPolicy JSON derived via deriving-aeson; wire format unchanged,
  the never-shipped nullary Administration decoder and its spec block removed
- Qualified aliases renamed to their modules

Also: nested condition/evaluation/notify monitor YAML (shared Pkg.MonitorConfig),
docs moved to monoscopehq.com, npm CLI packages removed, tests/golden -> test/golden.
2026-11-05 00:54:10 +01:00
Anthony Alaribe
16246fde0c One operation table for the CLI and MCP; granite charts, table-layout tables
- shared/src/Web/Operations.hs declares every /api/v1 operation once: MCP tool
  name, agent access, description, and its CLI command, arguments and listing
  columns. AgentCatalog.apiOperations and the CLI's per-resource command types,
  parsers and dispatch now derive from it; a generic parser and runCall replace
  them, and ResourceKind is gone. Typed Method/Verb, Input, Arg and Listing keep
  invalid argument shapes unrepresentable.
- Agent local tools: single-use KQL/tool helpers folded into executeLocal; tool
  names in errors come from the operation. get_code_context no longer builds a
  throwaway cache (fetchSourceEvidence takes Maybe Cache).
- Derived names: constructorName (Generic) for actionName/operationName,
  slackReadName via encodeEnumSC (SearchPersonal -> SearchPublicAndPrivate),
  StreamFrame ToJSON via deriving-aeson.
- CLI charts on granite (round ticks, exact x range, width-exact) and tables on
  table-layout (widest column absorbs overflow, single-line header rule).
- retry replaces hand-rolled backoff in retryOnDeadlock, retryOnPoolExhaustion
  and the CLI's withRetry; agoText replaces prettyTimeShort; http-types urlEncode
  in the CLI; text-metrics levenshtein replaces editDistance.

Net about -970 lines.
2026-11-03 22:35:58 +01:00
Anthony Alaribe
971221193d
CLI: UTF-8 output under any locale; --assert fails on undecodable responses (#782)
* CLI: write stdout/stderr as UTF-8 so help survives a C locale

Under LC_ALL=C (minimal Linux images) GHC's handles are ASCII, so help
pages with — or → printed the usage line and exited 1. The binary help
sweep now runs under LC_ALL=C and asserts the non-ASCII lines.

* CLI: metrics query --assert fails when the response does not decode

A response that isn't a metrics result skipped the assertion and exited 0,
so a CI gate built on --assert passed silently. It now exits 1.
2026-11-02 17:40:02 +01:00
Anthony Alaribe
2fb1c1e808
CLI: bulk ack works for log patterns; CLI specs as one lifecycle spec (#780)
* CLI specs: one spec of self-contained lifecycles

Merge CLIBinarySpec, CLIE2ESpec and CLILifecycleSpec into CLI.CLISpec: help
usage, error and list envelopes as tables, binary-vs-live-server walk, one
in-process lifecycle per workflow. withCliBin moves to CLI.Harness. The fake
router honours issues list filters and drops the /log_explorer and
/chart_data arms the CLI no longer calls.

* Log-pattern bulk actions accept ack like issues do

* CLI spec: one binary runner, per-example temp dirs for as-code yaml

* Drop dead telemetrySchemaJson; give the CLI binary spec a UTF-8 locale

The stripped test env dropped CI's LANG=C.UTF-8, so GHC's stdout fell back to
ASCII and help text containing — or → exited 1.
2026-11-02 15:24:57 +01:00
Anthony Alaribe
d5671d62ee
Unify autonomous agents across Slack, web and native jobs (#619)
Slack, web conversations, routines and native review/fix jobs share the capability catalog, current authority checks, run ledger and operation receipts. Runs, charts and saved reports open inside the conversation inspector.

Consolidates duplicate regression suites and documentation while retaining the product contracts, native recovery evidence and migration baseline transition.

Validation: all eleven exact-input local checks pass, including 1,216 integration examples (25 pending VM cases), 2,105 doctests, 315 unit examples, 16 CLI examples, 1,041 UI tests and 172 browser tests (one skip). Separate native/KVM acceptance is recorded in the review. The exact Linux AMD64 image was built locally on this Mac with OrbStack/Rosetta.
2026-11-02 11:15:00 +01:00
Anthony Alaribe
2f1419a968
Prepare CLI v0.6.31 and fix installed-version detection (#669) 2026-10-10 23:40:42 +02:00
Anthony Alaribe
a391e8e2cc
Fix CLI time windows, script safety, input validation, and help (#668)
* Fix CLI context windows and script safety

* Validate CLI inputs and run binary HTTP tests locally

* Consolidate CLI parsing and secure credential writes

* Auto-format code with fourmolu

* Simplify CLI redaction and enum parsing

* Fix custom CLI install directory example

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-10-10 23:04:17 +02:00
Anthony Alaribe
a460da14c8
Fix CLI resource lists and pick omitted IDs interactively (#662)
* fix(cli): show resource IDs and pick omitted resources

* Auto-format code with fourmolu

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-10-10 01:59:22 +02:00
Anthony Alaribe
908cbb9c20
Fix dashboard variable loading, CLI test events, and onboarding checks (#652)
* Fix dashboard variable loading, CLI test events, and onboarding checks

- Dashboards: charts init from a data-chart-config attribute instead of
  inline scripts; resolved constants travel with the swapped tab panel;
  variable picker filters via hx-live and syncs tagify without refetching.
- CLI: send-event/telemetrygen exchange a login session for the project's
  ingestion key (Ingestion effect, Credential sum type) and exit nonzero
  unless the span was exported.
- API: auth yields a typed ApiPrincipal (Authenticated | DemoReader);
  /api/v1/ingestion-key refuses the demo bypass.
- KQL isempty/isnotempty compare as text so numeric args don't coerce ''.
- Onboarding: no toast on automatic integration polls; CLI quick test
  sends one event; docs panel copy menu uses a native popover.

* Update page and e2e assertions for chart config attributes and lazy docs

* Auto-format code with fourmolu

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-10-07 03:01:22 +02:00
Anthony Alaribe
355a2a662e
Fix CLI status open issue count (#635) 2026-10-04 13:24:01 +02:00
Anthony Alaribe
5b8891a4c3
Add built-in observability routines (#585) 2026-09-19 23:09:05 +02:00
Anthony Alaribe
bf5d4a670a Lint the shared and CLI packages too, and fix what that turned up
CI ran `hlint src/`, so monoscope-shared and monoscope-cli — 22 modules, including
the KQL parser and the binary we ship to users — were never linted. shared/src was
already clean; the CLI had eight issues, one of them error severity.

All mechanical, all Relude preferences the rest of the codebase already follows:
T.pack over a String where toText does, T.unwords/T.unlines where Relude's are in
scope, the Data.Set alias the hlint config pins, and a lambda that is a <=< .

cabal test monoscope-cli:cli-tests passes 16/16 and the binary links.
2026-09-16 05:11:44 +02:00
Anthony Alaribe
ed0aebccf7
Support trace IDs in timestamped CLI lookups (#536) 2026-09-07 17:24:22 +02:00
Anthony Alaribe
6f91d5b6a4
fix(cli): preserve pipelines when adding shorthand filters (#518)
* fix(cli): compose shorthand filters as pipeline stages

* Preserve operator-free leading query stages
2026-09-07 04:26:38 +02:00
Anthony Alaribe
04cbfd4b3b
Fix CLI severity filters to match stored field and enum values (#514) 2026-09-07 02:40:57 +02:00
Anthony Alaribe
1f84715695
Honor explicit CLI time bounds without a default relative range (#510) 2026-09-07 01:03:10 +02:00
Anthony Alaribe
545ae6bf0c fix(cli): a bare search term is full-text again, not an unknown field
`monoscope logs search POISON_ROW_DROPPED` — the example in the command's own
--help and in docs/cli.md — has been exiting with

    invalid query: Unknown field "POISON_ROW_DROPPED"

since field validation landed in e7ad369. Nothing was wrong with the search
itself: `buildSearchParams` rewrites a bare term into `body has "X" or summary
has "X"` before it goes on the wire, and that is valid. But `validateEventsOpts`
validated `opts.query` — the raw term, before the rewrite — so the parser was
asked to resolve a search term as a column and the CLI exited before the rewrite
ever ran.

Validate the text that will actually be sent. Confirmed in ghci against this
tree: parseQueryToAST "chunkevt" is Left, and the rewritten form is Right.

Server-side is unaffected: the API only ever received the rewritten query, so
this was never visible through the web UI or a direct API call.

The two halves drifting apart is the actual defect, so the doctests on
rewriteBareQuery now assert that its output parses, not just what it looks like.
2026-08-12 11:41:29 +02:00
Anthony Alaribe
97f4ef845c feat(issues): acknowledging an issue silences it, and the silence has an end
"Acknowledged" promised nothing. It moved a row out of the Inbox and kept
sending alerts, so the only way to actually stop being paged was to archive —
which also meant "this isn't worth fixing". The word had no meaning you could
act on, and the Acknowledged tab was where issues went to be forgotten.

How the others resolve it: Sentry's Archive takes an explicit "until" (a
duration, or until it happens again N times) and stops notifying for it.
Datadog keeps monitor muting time-boxed and separate from status. PagerDuty's
ack is "I'm on it" with a timeout that re-pages if nothing happens.

The common thread is that silence is a promise with an expiry. So rather than
adding a second verb next to acknowledge, acknowledge *becomes* the silence
control — one concept, with a window:

  Open        needs triage                        notifies
  Acknowledged someone owns it, until <when>      silent, then back to Inbox
  Archived    not actionable                      silent, stays hidden

- apis.issues gains acknowledged_until (0125). A far-future sentinel — the same
  one monitors use for an indefinite mute — encodes "until it regresses".
  It supersedes cooldown_until, which said the same thing but only for
  log_pattern_rate_change and only for a fixed 24h.
- The notification paths honour it. claimIssueNotification, the new-endpoint
  claim, and the error-subscription sweep all now skip acknowledged and
  archived issues; a lapsed ack stops blocking immediately rather than waiting
  on the sweep. This is the actual bug: none of them looked at ack state before.
- Detectors consult isSilenced before firing a fresh issue for a target, so an
  ack means "don't tell me about this again", not "hide the old row".
- expireLapsedAcks (in the 10-min notification sweep) returns timed acks to the
  Inbox with an ack_expired activity event, so one can't rot out of sight.
- UI: the acknowledge control is a split button — one click silences until it
  regresses, the caret offers 1h/4h/8h/1d/3d/1w. Rows and the detail header
  show the time left; each tab says in one line what it means for notifications.
  Bulk actions now offer only the transitions that make sense from the tab
  you're on. Ack and archive were <a> without href: not keyboard-operable.
  They are buttons now, and the icon-only row actions have accessible names.
- API/CLI: POST /issues/:id/ack?duration_minutes=, issues bulk duration_minutes,
  monoscope issues ack --for MINUTES. Absent duration = indefinite, which is
  what ack did before, so existing callers keep their semantics.

The duration menu and its "4h left" label are shared with monitor muting, so
"for how long?" is answered the same way in both places.
2026-08-12 01:00:19 +02:00
Anthony Alaribe
593eb63e52 Fix the trace waterfall's parent linking, logfmt escaping, and the 90d scan
Found by actually running the commands end-to-end:

- The waterfall keyed the tree on the row UUID, but parent_id holds a *span*
  id (the server aliases context___span_id to latency_breakdown). Every span
  therefore looked like a root and nothing was ever indented. EventRow gains
  spanId and the tree keys on it. The test fixture had encoded the same
  misunderstanding, so it passed throughout — it now carries both ids with the
  real linkage.
- logfmt emitted the summary's ANSI colour codes and did not escape quotes or
  newlines, so a log body containing either split one event across lines. The
  summary is stripped of styling and values are escaped.
- events get --tree hardcoded a 90d scan, which the store can refuse outright
  ("merge-on-read dedup exceeded its per-query limit"). --since narrows it;
  the default is unchanged.
2026-08-06 16:46:48 +02:00
Anthony Alaribe
5d47e49b1c Distillation pass over the new CLI code
Removes three forks of the same logic and turns on the tests that would have
caught them:

- CLI.Chart.seriesFrom generalises seriesFromMetrics over any header/grid pair,
  so CLI.Dashboard.widgetSeries and its private isTimeCol go away. The two had
  already drifted (one recognised _timestamp, the other didn't). CLI.Commands's
  third copy of the time-header list goes with them.
- Pkg.CLIFormat gains extractColIdxMap and renderSummaryCell; CLI.LogView and
  CLI.Commands each had a byte-identical private copy of both.
- LogView measured ANSI width with T.filter (/= '\ESC'), which leaves the
  '[2m' behind and under-sized the message column whenever colour was on. It
  now uses CLI.Chart.visibleWidth, which the module already imported.

Dead weight: ChartStyle was never set to anything but Line, so the type, the
field and the unreachable canvas fill are gone; clamp/clampD collapse into
Data.Ord.clamp; two zipWith/foldl' plumbings threaded an index nobody read.
apiDashboardData stops returning (widget, window) tuples — the window comes
from the request, which also means a dashboard with no widgets reports its
range instead of null.

The doctest runner now covers cli/CLI (768 -> 820 examples). Turning it on
found two of my own examples asserting things that were never true, one that
would not compile, and two list literals that were ambiguous under the
runner's OverloadedLists. Four hspec cases duplicating doctests are deleted.
2026-08-06 11:02:15 +02:00
Anthony Alaribe
45b270786b Use a filter the store can plan for the status errors panel
`errors == true` fails to plan on TimeFusion (the column is Utf8View, so the
boolean comparison has no common type). status_code=="ERROR" is the filter
that actually works against both stores.
2026-08-06 03:08:08 +02:00
Anthony Alaribe
ebd5759a1a Drop helpers left orphaned by the new renderers 2026-08-06 03:06:30 +02:00
Anthony Alaribe
1a009d445c Give events context the same --format as search and tail 2026-08-06 03:03:51 +02:00
Anthony Alaribe
b353d1051d Lay out help examples as a list instead of a reflowed paragraph
optparse's plain footer reflows its string, so every command's Examples block
collapsed into an unreadable wall of run-on commands that could not be
copy-pasted. examplesFooter lays the lines out explicitly.

Also folds the CLI's two copies of "how wide is the terminal" into
CLI.Table.termWidth, which already guards the stdin probe.
2026-08-06 03:02:31 +02:00
Anthony Alaribe
67cfcb871d Add monoscope status and monoscope open, and fix summarize-by
Two things a terminal-first user reaches for that were missing:

- `monoscope status` — one screen answering "is anything wrong right now":
  throughput, errors, busiest services, open issues, alerting monitors. A panel
  whose query fails says so inline; an unavailable panel and a quiet one are
  not the same thing.
- `monoscope open <target> [id]` — the web UI link for a trace, issue,
  dashboard, log query or the project, opened or just printed. The host comes
  from /api/v1/me (new `host_url` field) rather than being guessed from the API
  URL, so self-hosted and split API/UI hostnames both work.

Writing status surfaced three real bugs in `summarize <agg> by <field>` with no
time bin — the shape behind `chart --type bar` and the
`summarize count() by resource.service.name` example in our own docs. It failed
for every caller:

1. The grouping column was never projected, so the query returned a bare column
   of numbers with nothing saying which group each belonged to, and no decoder
   accepted it. It now emits (group, aggregate), matching the binned branch.
2. /api/v1/metrics defaulted to the DTMetric decoder, which needs a leading
   timestamp. Charts.decoderFor now infers the decoder from the query shape:
   no `by` → scalar, `by bin(...)` → timeseries, anything else → label/value.
3. `| sort by count_ desc` was silently dropped (the branch hand-rolled its
   ORDER BY), and once aggregates were cast to text for the decoder it would
   have sorted them lexicographically. ORDER BY now resolves aggregation
   aliases back to their expressions, so it sorts numerically.

Also: test-dev and the unit/integration suites needed the new packages on their
dependency lists.
2026-08-06 02:58:53 +02:00
Anthony Alaribe
e684da1374 Document the terminal charts, dashboards and log formats
docs/cli.md gains sections for `chart`, `dashboards render`, the trace
waterfall, --format, and the dashboard-render JSON envelope; the install
section now states the no-prerequisites guarantee and what enforces it.
monoscope-skills learns the same commands, with the agent-facing framing:
render a dashboard to find broken or empty widgets, sort a trace's spans by
duration rather than reading the waterfall, and bound `tail` with `timeout`
so a tool call cannot hang.

Two bugs found while writing the examples:
- chart ignored --json and always drew; it now honours the output mode.
- a failed query came back as an empty result set with `error` set, which the
  CLI rendered as "no data in range" and exited 0. It now prints the server's
  message and exits non-zero in every mode.
2026-08-06 02:04:07 +02:00
Anthony Alaribe
7c01233ed6 Slim the CLI binary and rebuild its release pipeline around it
The CLI no longer needs the frontend build, proto codegen, or the eight
Homebrew packages the macOS job used to install — none of that is reachable
from monoscope-cli. Removing openapi3 from the shared package (a CLI has no use
for schema generation; the ToSchema instances move to Pkg.DeriveUtils and the
new Web.WireSchemas) takes the stripped -O2 binary from 107MB to 97MB.

- cli-release.yml: builds monoscope-cli:exe:monoscope only, with -O2
  -split-sections plus --gc-sections/-dead_strip and strip.
- scripts/check-cli-linkage.sh fails the build if the binary links libpq,
  librdkafka, grpc, protobuf or a compression library — the no-prerequisites
  promise is now enforced rather than assumed. It also rejects /opt/homebrew
  paths, and the macOS job repoints Homebrew's zlib at /usr/lib and re-signs,
  which would otherwise have shipped a binary that only runs on a Mac with brew.
- cli-tests now run in the main CI workflow; make cli-build/cli-test/
  cli-release-build for local use.

Measured but not taken: dropping brick/vty saves only 4MB, which does not pay
for losing the arrow-key project picker at login.
2026-08-06 01:49:06 +02:00
Anthony Alaribe
7f5fdf27c7 Make logs and traces read like logs and traces in the terminal
- One event per line (time, severity, service, duration, message, trace id)
  instead of a bordered table, which is what you actually scan during an
  incident. --format line|table|logfmt picks the shape; --json/--yaml still win.
- events get --tree draws a waterfall: spans indented by depth with a bar
  showing where each sat inside the trace, so "what took the time" is visible.
  Spans whose parent is missing render as roots rather than disappearing.
- tail gets --format, --interval and --since (backfill), and now overlaps its
  poll window so a row written a second late is not skipped forever. --grep
  matches the message, span name and service rather than the raw joined row.
- New cli-tests suite covers the renderers as pure [Text]: widths, empty and
  flat series, ANSI-safe padding, waterfall shape, dashboard band layout.
2026-08-06 01:35:40 +02:00
Anthony Alaribe
2e45515019 Render dashboards, widgets and ad-hoc charts in the terminal
Adds GET /api/v1/dashboards/{id}/data: the server resolves constants,
variables and every widget's query up front and returns the 12-column grid,
so a non-browser client draws the dashboard without re-implementing any of it
or firing a request per widget.

CLI side:
- CLI.Chart draws braille line charts, block bar charts, sparklines and stat
  tiles as pure [Text], with nice y-ticks, time-formatted x labels and a legend.
- CLI.Dashboard maps the grid onto the terminal width, so widgets that sit side
  by side in the browser sit side by side here; group widgets nest a sub-grid.
- monoscope dashboards render <id> [--tab] [--widget] [--var k=v] [--watch]
- monoscope chart '<kql>' [--type line|bar|stat|table] [--watch]
- metrics chart now goes through the same renderer instead of a bare sparkline.

Widget.chartQuery moves the browser's default aggregation server-side, and
getTerminalSize is no longer called when stdout is not a terminal (it reads
stdin, which threw EOF under a pipe).
2026-08-06 01:27:51 +02:00
Anthony Alaribe
26a0d78df8 Split the CLI into its own package so it no longer needs libpq
The monoscope binary linked lib:monoscope, which drags in libpq, librdkafka,
grpc, protobuf, snappy, lz4 and zstd. Anyone installing just the CLI had to
install all of them first.

- New monoscope-shared package holds the DB-free modules the server and CLI
  genuinely share (wire types, KQL grammar, HTTP effect, formatting).
- New monoscope-cli package owns CLI.*; it depends on monoscope-shared only.
- Pkg.Deriving splits the pure DerivingVia wrappers out of Pkg.DeriveUtils
  (which re-exports them, so no call site changes).
- Web.Wire homes Paged / device-flow / FacetValue away from their DB-bound
  modules.
- runCLILifecycle moves into the integration suite, which can see both packages.

The produced binary now links only libSystem, zlib, ncurses, iconv and ffi.
2026-08-06 01:17:03 +02:00
Anthony Alaribe
a5a015f3fd
Cli lifecycle as code (#428)
* feat(cli): monitors/dashboards-as-code + in-process CLI lifecycle tests

- Move cli/ into the library (CLI.Main.cliMain); exe is a thin Paths wrapper.
  Enables runCLILifecycle: real optparse parse + command pipeline in-process,
  HTTP routed to handlers via runHTTPtoServant, stdout captured (silently).
- monitors apply was not an upsert: an id-less file created a new row on
  every apply. Add POST /api/v1/monitors/apply (natural key: alert title,
  mirroring dashboards' file_path), wire CLI applyOne to it.
- Add monitors yaml ID dump (inverse of monitorFromInput) so dumps round-trip
  through apply; same contract dashboards already had.
- Test router: thread verb + query params through writes (PUT/PATCH/POST were
  indistinguishable, mute --for lost its duration); add monitors, dashboards,
  events, facets, metrics, share routes + DELETE routing.
- CLILifecycleSpec: monitors-as-code and dashboards-as-code lifecycle walks
  (apply idempotency, yaml round-trip, mute/unmute/star/duplicate/delete).

* feat(cli): single-envelope chunked search, OTLP/HTTP ingestion, scalar metrics fix + lifecycle tests

- Chunked search no longer leaks slices as NDJSON: slices fetch newest-first
  with the remaining --limit budget, rows dedupe by id at boundaries, and the
  result renders through the same single-envelope path as unchunked searches.
  has_more/cursor come from the stopping slice. --no-chunk removed;
  --chunk-hours 0 disables slicing.
- OTLP/HTTP ingestion (POST /v1/traces, /v1/logs, application/x-protobuf)
  reusing the gRPC processing pipeline with x-api-key header auth. The
  hs-opentelemetry exporter is http/protobuf-only, so send-event/telemetrygen
  had no working collector-less path; configureOtelEnv now also honors a
  pre-set OTEL_EXPORTER_OTLP_ENDPOINT. Handlers injected into Routes.server
  from System.Server (proto-lens orphan IsLabel instances clash with
  generic-lens labels if Routes imports OtlpServer).
- Scalar metrics queries (summarize with no by-clause) crashed on float8/Int
  decode under the default DTMetric pivot; queryMetrics now infers DTFloat —
  fixes the documented 'metrics query ... --assert' CI-gate flow.
- Lifecycle tests: investigation pipeline (search → get → context →
  share-link resolving unauthenticated via shareLinkGetH), chunked
  single-envelope contract (ordering, dedupe, --limit/has_more), metrics
  --assert exit codes, and send-event shipping a span over the real OTLP/HTTP
  wire into a live Warp instance (withLiveServer).

* Auto-format code with fourmolu

* lint: bring src/CLI under hlint zero-hint policy

Moving cli/ into src/ exposed it to 'hlint src' for the first time.
toText over T.pack, putTextLn, pass, universe, fromLeft, whenNothingM_,
Relude decodeUtf8 (already lenient), S alias for Data.Set, eta reductions.

* Auto-format code with fourmolu

* refactor(cli): address PR #428 review feedback

- warn (printDebug) + document when chunked-search dedup falls back to
  full-row JSON keys because the response lacks an id column
- note why DashUnstar can't use runLifecycle (server route is DELETE
  /star, no POST unstar endpoint)
- convert IssueList/EndList/MemberAdd positional Maybe fields to records

* fix(make): TEST_MATCH was silently matching nothing

GHCi :main does not strip mid-word quotes, so --match="PATTERN" passed
literal quote chars to hspec and every filtered run reported
'0 examples, 0 failures'. Pass --match PATTERN unquoted; for patterns
with spaces put the quotes in the variable: TEST_MATCH='"CLI lifecycle"'.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-06-11 23:05:13 +02:00
Anthony Alaribe
60bab8768a fix: CLI auth honors output mode for C7 guard; update stale integration specs
- cli: AuthCmd never called resolveMode, so isJsonOutput stayed False and
  the C7 non-interactive guard never refused the device flow when piped
- LogSpec: processMessages now returns Either WriteFailure (acks, poison);
  length on the Either was always 1
- ApiV1Spec: get_schema serves Schema.deriveSchema over the live column
  set, not the static telemetrySchema
2026-06-11 14:55:36 +02:00
Anthony Alaribe
2d18564cda docs/comments: drop competitor name from prose
Neutralises descriptive references to another vendor in comments and
user-facing copy. Functional behaviour is unchanged.

- cli/CLI/Table.hs, cli/CLI/Core.hs, cli/CLI/Commands.hs: comment polish
  around the table renderer + bareword search.
- src/Web/MCP.hs: trim vendor list in tool-naming convention comments.
- src/Pages/Anomalies.hs: rephrase breadcrumb-source comments. The wire
  attribute keys themselves (sentry.breadcrumb.category/type/message)
  are SDK-defined identifiers and stay — those are what we parse.
- src/Pkg/ErrorFingerprint.hs: drop the analogy in normaliser comments.
- src/Pages/Onboarding.hs: rewrite the FAQ answer (also fixes two typos
  and an awkward sentence).
- docs/mcp.md: simplify the verb-first naming-convention sentence.
2026-06-09 22:12:58 +02:00
Anthony Alaribe
62502d73b3 cli + schema: rework output modes, bare-string search, structured errors, auto-chunk, multi-service, schema introspection, facets fallback
CLI
- Mode model: drop --agent and --output FORMAT; flags are now --json | --yaml | --table, plus auto-detect (TTY → table, pipe → JSON). MONOSCOPE_FORCE_COLOR overrides.
- Pretty tables: new Sentry-style Unicode box-drawing renderer (cli/CLI/Table.hs) with per-cell CellStyle/Severity colouring, terminal-width truncation, and a pagination cue under list output. Listings (issues, monitors, dashboards, api-keys, teams, members, endpoints, log-patterns) flow through it via runListVia.
- Bare-string queries → full-text: foldFiltersIntoQuery rewrites a bareword/phrase to `body has "X" or summary has "X"` so `monoscope logs search POISON_ROW_DROPPED` works instead of 42703.
- --service repeatable: 0/1/N values expand to noop / == / `in (...)`; parser already supports `in`.
- --level normalises to upper-case.
- Auto-chunk long --since: events|logs search splits windows > --chunk-hours (default 1h) into slices anchored on a single now, streams NDJSON per slice, short-circuits on --first/--id-only, --no-chunk opts out.
- events get --field FIELD / --show-body: project subset of the event JSON without forcing jq.
- --limit alias added across listings (alongside --per-page / --top).
- Friendly gateway errors: 502/503/504 + Cloudflare HTML are collapsed to one-liners with a "narrow --since" hint; one jittered retry on transient 5xx + ResponseTimeout.
- Strict stderr in non-table mode: printError emits NDJSON {"error": {...}} on stderr; ANSI suppressed everywhere when stdout isn't a TTY.
- --debug sets X-Debug:1 header so the server can include raw SQL/Hasql exception text in error.details.
- renderAPIError decodes structured server errors and shows just the one-liner.

Server
- Schema introspection (C1): startup queries information_schema.columns for otel_logs_and_spans, derives the dotted-form attribute set, seeds Pkg.Parser.Expr.flattenedOtelColumnsRef (via setFlattenedOtelColumns). flattenedOtelAttributes reads from the cache; falls back to flattenedOtelAttributesBuiltin if introspection fails.
- /api/v1/schema reads from the live cache via Models.Telemetry.Schema.deriveSchema, which decorates each live column with the hand-coded FieldInfo when present and drops dotted-form Schema.hs entries that don't match a real column.
- Facets fallback (C2, new Web.FacetsFallback module): when apiFacets gets an empty result for a requested field, run a bounded SELECT col, COUNT(*) FROM otel_logs_and_spans … GROUP BY col LIMIT 50 on the whitelisted column. Tags response with "source": "fallback". Module is split out so Hasql.DynamicStatements imports don't perturb Generic derivation in Web.ApiHandlers.
- Structured KQL errors (A2): queryEvents returns JSON-shaped 400s with {error: {code, message, field?, suggestion?, details?}}. Detects column-not-exist for both Postgres (column "X" does not exist) and TimeFusion (Schema error: No field named X).
- Severity field aliases: severity.text, severity.severity_text, level all canonicalise to severity.severity_text.
2026-06-09 19:28:51 +02:00
Anthony Alaribe
7a4b94cf96
ingestion: TF-strict commit, per-partition offsets, row-level bisection (#402)
* ingestion: TF-strict commit, per-partition offsets, row-level bisection

The 2026-06-01 TimescaleDB outage exposed several ways the consumer
advances Kafka offsets past failed batches, losing hours of customer
data. Commit semantics, the dual-write asymmetry, and the fire-and-forget
DLQ all combined to step over any failure that wasn't a hard exception.

- kafkaService now commits per (topic, partition) via
  commitPartitionsOffsets. Only groups whose batch succeeded -- or whose
  poison content was successfully DLQ'd -- contribute offsets. Failed
  groups leave their partitions uncommitted; lag growth becomes the
  outage signal. DLQ is no longer subscribed by the live loop.
- pubsubService mirrors: poison -> DLQ-then-ack; transient -> no ack.
- publishToDeadLetterQueue returns Either Text () so the caller can gate
  commit on DLQ success.
- bulkInsertOtelLogsAndSpans bisects on non-transient Hasql failures,
  isolating a single poison row in O(log N) attempts so a bad row no
  longer wedges the partition.
- bulkInsertOtelLogsAndSpansTF requires TF success to return normally;
  PG-only failure is logged via PG_WRITE_FAILED_TF_OK and tolerated (PG
  is being phased out).
- New Pkg/IngestError classifier replaces the brittle string-match
  isUnrecoverableError. Unknown errors default to PoisonBatch
  (DLQ-actionable) rather than TransientInfra (silent stall).

* Auto-format code with fourmolu

* ingestion: inline transient-error check, tighten review nits

PR-review polish on top of the durability changes:

- Inline the Hasql-transient check at the two failure-handler call sites
  in Pkg/Queue.hs and delete Pkg/IngestError.hs. Two 1-line predicates
  with two call sites in one file wasn't worth its own module.
- Drop the now-trivial allTopics = kafkaTopics alias; inline into
  consumerSub and drop the redundant all_topics startup-log field.
- publishToDeadLetterQueue: collapse the case lefts results to
  maybe (Right ()) Left . listToMaybe . lefts.
- Lower the bisection depth cap from 12 to 10 (still 1024 rows, well
  above chunkSize=700). Log BISECT_DEPTH_EXHAUSTED before re-raising
  so the theoretical stall is visible if it ever happens in practice.
- Split bulkInsertOtelLogsAndSpansTF's (_, Left eTf) wildcard into
  explicit (Right _, Left eTf) and (Left ePg, Left eTf) arms so a
  both-failed event logs both errors instead of swallowing the PG side.
- Replace the CLAUDE.md cross-reference in the dual-write doc with an
  inline note; CLAUDE.md is not in the repo and would rot.

* Auto-format code with fourmolu

* ingestion: extract isTransientException, drop commitTPs IORef

Second round of review polish on the durability changes:

- Data/Effectful/Hasql: export `isTransientException :: SomeException
  -> Bool`. Collapses the four `maybe False isTransientHasqlError
  (fromException e)` repetitions across Queue.hs and Telemetry.hs to
  `Hasql.isTransientException e`.
- Pkg/Queue.hs:kafkaService: drop the `commitTPs` IORef +
  `recordSuccess` writer; restructure the topic-group loop as
  `fmap concat $ forM ...` that returns the per-batch TopicPartition
  list directly. `tpsFor` becomes a small pure helper. Removes
  `newIORef`/`modifyIORef'`/`readIORef`/`liftIO` ceremony.
- Pkg/Queue.hs:publishToDeadLetterQueue: replace `listToMaybe . lefts`
  with `partitionEithers + listToMaybe`, matching the existing idiom
  used at the `pollMessageBatch` site in the same file.
- Pkg/Queue.hs: add an inline comment by the (now-absent) poll
  threadDelay site noting why no backoff lives there, so future
  readers don't reintroduce one as a "fix".

* ingestion: concatMapM, lefts, $>, trim doc

Idiomatic touch-ups on top of the durability changes:

- Pkg/Queue.hs:kafkaService: replace `fmap concat $ forM` with
  `concatMapM` from Control.Monad.Extra. Lift the per-group action
  to a `let processGroup = ...` binding so the layout closes cleanly.
- Pkg/Queue.hs:publishToDeadLetterQueue: drop the partitionEithers
  pair-with-ignored-second; use `lefts` directly.
- Models/Telemetry/Telemetry.hs:bulkInsertOtelLogsAndSpans: switch
  the POISON_ROW_DROPPED branch from `0 <$ action` to `action $> 0`
  so it reads left-to-right.
- Data/Effectful/Hasql.hs: trim the isTransientException doc comment
  to a single WHY line.

* Auto-format code with fourmolu

* ingestion: hoist snippet build, document edge cases, guard sentinels

Address three correctness/observability issues and two conciseness
nits surfaced in review:

- Models/Telemetry/Telemetry.hs:bulkInsertOtelLogsAndSpans: hoist
  otelRowSnippet above the bisection recursion so each row's snippet
  is built (and any OTEL_UNPARSEABLE_IS_REMOTE log fires) exactly once
  even when bisection recurses O(log N) levels. Bisection now operates
  on a vector of (record, snippet) pairs.
- Models/Telemetry/Telemetry.hs:bulkInsertOtelLogsAndSpans: docstring
  notes the non-TF-mode duplicate-row edge case when a mid-bisection
  transient propagates and Kafka redelivery re-mints UUIDs. The
  hypertable has no UUID-only unique constraint we can ON CONFLICT
  against; under TF-strict semantics PG failures never trigger retry,
  so the bug only surfaces in tests / legacy.
- Pkg/Queue.hs:publishToDeadLetterQueue: collapse the explicit
  results-binding + lefts/listToMaybe chain to
  `fmap sequence_ $ forM ...`; same first-Left semantics, less syntax.
  Docstring notes the duplicate-in-DLQ-on-partial-failure tradeoff.
- Pkg/Queue.hs:kafkaService:tpsFor: filter out negative offsets in
  the partition-commit map so a librdkafka sentinel (OffsetEnd = -1,
  OffsetInvalid) can't rewind the consumer to the start. Shouldn't
  appear in pollMessageBatch output but making the assumption explicit.

* ingestion: clarify DLQ aggregation, drop unreachable guard, comment <*>

Review polish, fourth pass:

- Pkg/Queue.hs:publishToDeadLetterQueue: switch back to explicit
  `results <- forM ...; pure (sequence_ results)` instead of
  `fmap sequence_ $ forM ...`. Same semantics, reads left-to-right
  and matches the explicit-bind idiom used elsewhere in this file.
- Models/Telemetry/Telemetry.hs:bulkInsertOtelLogsAndSpans: drop the
  unreachable `go _ pairs | V.null pairs` guard. The outer guards
  V.null records; the single-row branch handles 1; recursive calls
  always split a >=2 vector — so 'go' never sees an empty vector. A
  comment notes the reasoning.
- Models/Telemetry/Telemetry.hs:bulkInsertOtelLogsAndSpans: comment
  the sequential `<*>` semantics so a future refactor doesn't quietly
  introduce parallelism that would let the right half attempt rows
  after the left half throws (changing the transient + depth-exhaust
  semantics).

Note: HM.fromListWith was considered for tpsFor to match the rest of
Queue.hs, but Kafka.Types.PartitionId has Ord but not Hashable, so
Data.Map.Strict stays.

Tracking deferred tests in #403.

* ingestion: trim docstring, where-bind rawInsert chunks, uncurry tpsFor

Three review nits:

- bulkInsertOtelLogsAndSpans: trim the ~25-line docstring to a 5-line
  summary. The non-TF-mode edge case and the snippet-hoisting rationale
  are PR-discussion material, not per-call doc.
- bulkInsertOtelLogsAndSpans:rawInsert: move chunkSize / chunks /
  chunkStmt from `let ... in case` to a `where` clause so the body
  reads case-first.
- kafkaService:tpsFor: deduplicate the per-partition K.TopicName topic
  by mapping `uncurry (K.TopicPartition (K.TopicName topic)) . second
  K.PartitionOffset` over Map.toList. Same result, no constructor
  repetition.

Skipped: HM.fromListWith for tpsFor — Kafka.Types.PartitionId has Ord
but not Hashable (confirmed via :info), so Data.Map.Strict stays.

* ingestion: whenLeft_ tfThread, name firstFailure, guard bisect cap

- bulkInsertOtelLogsAndSpansTF tfThread: switch the Left/Right case
  to whenLeft_ \e -> do ... (already 8+ call sites in the codebase).
  Drops two lines and matches local idiom.
- publishToDeadLetterQueue: name the aggregated result 'firstFailure'
  with an inline comment so the sequence_ short-circuit semantics
  (first Left wins, subsequent failures only in the dlq log above)
  is explicit at the call site.
- bulkInsertOtelLogsAndSpans: relax docstring to tie the bisect cap
  (10 levels → 1024-row ceiling) to messagesPerPubsubPullBatch /
  kafkaBatchSize rather than chunkSize. Add a runtime
  BISECT_CAP_MAY_NOT_ISOLATE log on inputs above 1024 so on-call sees
  the bound being approached before BISECT_DEPTH_EXHAUSTED fires on
  a real poison row.

* ingestion: simplify firstFailure / tpsFor / processGroup

- publishToDeadLetterQueue: drop the intermediate `firstFailure` binding;
  inline as `pure $ sequence_ results`. The short-circuit comment now
  sits above the pure.
- kafkaService: drop `concatMapM` import (used exactly once); use
  `concat <$> mapM processGroup ...`. Merge the two adjacent let blocks
  inside the do (byTopic + processGroup) and lift `tpsFor` to the outer
  `where` clause — it doesn't close over anything inner.
- kafkaService:tpsFor: rewrite as a list comprehension. Easier to read
  than the point-free uncurry+second form.
- bulkInsertOtelLogsAndSpans: drop the unnecessary `Relude.when`
  qualifier — Relude is imported unqualified.

* Auto-format code with fourmolu

* ingestion: concatMapM, demote bisect-cap log, pattern-guard rawInsert

- kafkaService: switch `concat <$> mapM` back to `concatMapM` from
  Control.Monad.Extra (extra is already a build-dep).
- bulkInsertOtelLogsAndSpans: demote BISECT_CAP_MAY_NOT_ISOLATE from
  logAttention to logInfo. It is a capacity advisory, not an error;
  on-call shouldn't get paged during normal high-volume ingestion.
- bulkInsertOtelLogsAndSpans:rawInsert: switch the single-chunk vs
  multi-chunk branch from `case chunks of` to a pattern guard
  (`| [single] <- chunks = ... | otherwise = ...`). Shorter and uses
  the LambdaCase/ViewPatterns extensions already enabled.

* ci: fix hlint warnings in SchemaCatalog

- upsertSummary: `tryAny ... >>= either (const pass) (const pass)`
  collapses to `void $ tryAny ...` (hlint Use whenLeftM_).
- Drop now-unused `Data.Text qualified as T` import.

* cli: adapt to hs-opentelemetry-api API changes

New hs-opentelemetry-api added arguments to two signatures:
- shutdownTracerProvider :: TracerProvider -> Maybe Int -> ... (timeout)
- TracerOptions :: Maybe Text -> [ExceptionHandler] -> TracerOptions

Pass Nothing / [] at the existing call sites.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-06-02 12:59:16 +02:00
Anthony Alaribe
bc9bca6ab3 refactor(queue,otlp): tighten queue/otlp pipeline logging and time tracking
- Replace ad-hoc liftIO getCurrentTime + manual diff with Log.timeAction in
  processBatchPipeline and processList; drop redundant local bindings.
- Collapse per-call runLogT scaffolding in kafkaService into a single
  LogT/localData context so client_id/group_id/topic ride along automatically.
- Group kafka poll results by topic before dispatch so multi-topic polls get
  the right ce-type and attributes per batch.
- Move kafkaProducerProps under getOrInitKafkaProducer (only caller).
- publishToDeadLetterQueue now logs publish failures instead of silently
  discarding them; threads appLogger through pubsub/kafka callers.
- Simplify Start.startApp tracer bracket.
2026-05-13 15:42:36 +02:00
Anthony Alaribe
8dd212a493
Fix/events search descendants only (#373)
* fix(events): default search to exact predicate match, scope expansion to descendants

Events search used to return every span in any matching trace — siblings,
parents, and uncles included — so a query like `kind=="server" AND
sdk_type=="JsExpress"` would also pull in pg.query, pg-pool.connect, and
internal spans that failed the predicate. Two fixes:

1. Default to exact match. New `with_children` flag (POST body field on
   /api/v1/events/query, query param on GET /api/v1/events) opts back into
   trace context. CLI exposes `--with-children`. UI handler still passes
   True to preserve the LogExplorer waterfall.

2. When expansion is on, return only descendants of matched rows. The
   broad SQL fetch in selectChildSpansAndLogs is preserved for performance
   (one indexed scan on the partitioned otel_logs_and_spans), and a new
   keepDescendantsOf BFS trims the candidate set to actual sub-trees of
   the predicate hits. The function previously returned the whole trace
   despite its name.

Caught a latent bug along the way: selectChildSpansAndLogs built its
column-index map from getProcessedColumns output, which retains
"<expr> as <alias>" strings — so lookups for "latency_breakdown" /
"parent_id" silently returned Nothing. Piped through listToColNames so
the alias-only keys match.

* Auto-format code with fourmolu

* fix(hlint): drop lambda in keepDescendantsOf BFS filter

* review: tighten keepDescendantsOf

- Drop Data.Map.Strict; reuse the already-imported HashMap (HM is O(1)
  avg, and avoids a new import surface for one helper).
- Fix latent bug: rows missing a span_id (latency_breakdown) used to pass
  the visited-set filter unconditionally via `maybe True`, leaking
  unidentifiable rows into descendants. Flip to `maybe False` — drop
  them, since we can't dedup or descend from them.
- Switch BFS to prepend-then-reverse to avoid O(n²) `acc <> newKids` /
  `rest <> newSids` chains that materialise quadratic spines under lazy
  evaluation.
- Cache `V.toList seedSpanIds` (used twice).
- Use `<$ guard` for the CLI flag — matches the surrounding style for
  Maybe-shaped param entries.
- Clarify the `count` vs `logsData length` assertion in the LogSpec
  trace-tree test.

* test(log-explorer): real cursor-pagination regression test

The existing 'should paginate through multiple pages using cursor' test
fetches page 1 and asserts hasMore=True, but never actually fetches
page 2 — so a broken cursor would silently pass. It also ingests 1000
messages all at frozenTime, so any ordering bug across the cursor
boundary would be hidden by ties.

New test walks pages 1 → 2 → 3 from 5 rows ingested at distinct
timestamps with `| limit 2`, asserting:
  * each page returns the exact expected slice (idx [1,2], [3,4], [5])
  * the -0.001s cursor nudge actually skips the boundary row (no
    overlap between consecutive pages)
  * hasMore flips False on the last page

Identifies rows by querying `attributes.page.idx` from the DB after
extracting `latency_breakdown` (= context___span_id) from each result
row — LogResult doesn't surface attributes directly.

* test(log-explorer): guard cursor advancement and pageIdsFor shape

* review: address keepDescendantsOf review notes

- Rename bfs → go (the queue prepended new nodes, making it DFS in
  practice; correctness is unaffected but the name was misleading).
- HM.findWithDefault [] over fromMaybe [] (HM.lookup ...) for the
  parent→children lookup, matching the library's intended API.
- Drop two over-explanatory inline comments inside keepDescendantsOf
  (the maybe False rationale and the prepend-build-then-reverse note);
  Haddock already covers what's load-bearing.
- Document in Haddock that seeds are not included in the output —
  callers already hold them in requestVecs.
- Note on the V.null seedSpanIds early return: kept to skip the SQL
  round-trip; comment why so the redundancy with keepDescendantsOf's
  guard reads as intentional.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-05-07 23:47:31 +02:00
Anthony Alaribe
760779b345 fix(cli): preserve JSON types and clean summary in events search output
Numbers, bools and arrays were being stringified in the events search
JSON envelope, and the summary column came back as a JSON-encoded string
of `field;style⇒value` markup. Agents/jq consumers had to coerce every
cell. Now rawRows preserve types end-to-end, empty strings collapse to
null, and summary/latency_breakdown are run through cleanSummaryValue.
2026-05-05 15:15:42 +02:00
Anthony Alaribe
92d5dfa710 feat: add send-event and telemetrygen CLI commands
Adds two new commands:
- `send-event -m TEXT` — sends a real OTel span from scripts/CI; supports
  --level, --service, --tag, --extra, --resource flags
- `telemetrygen --kind=trace|log|metric` — generates synthetic load at a
  configurable rate; replaces the external `go install telemetrygen` workflow

Both commands derive the OTLP gRPC endpoint from the configured API URL
and authenticate automatically via the stored API key.

Shared helpers (withOtelProvider, configureOtelEnv, sendSpan) eliminate
duplication between the two commands.

Updates README, docs/getting-started.md, docs/DOCKER_COMPOSE_README.md,
and the onboarding modal to use the new commands instead of the external
binary with manual API key headers.
2026-05-03 22:52:01 +02:00
Anthony Alaribe
4119ccb8ca
Improve mcp and cli (#368)
* fix: surface OTel exception events in KQL search and issue stack_trace

- KQL: `attributes.exception.{type,message,stacktrace,escaped}` now
  COALESCEs the flattened column with a jsonb_path probe over span events
  (`event_name=="exception"`). Previously every search returned 0 because
  hs-opentelemetry records exceptions as span events, not flat attributes.
- API: `apiIssueGet` synthesises a span-chain "logical stack" when the
  source `exception.stacktrace` is empty (true for 100% of monoscope's
  prod exceptions — GHC backtraces aren't captured by default).
- Pages.Telemetry: guard `renderSpanRecordRow` integer divisions against
  divide-by-zero (empty service group / all-instant trace) — root cause
  of the recurring `ArithException` in the issue detail page.
- CLI: bump HTTP response timeout 30s -> 5min so long KQL searches
  don't surface as `ResponseTimeout` errors.

* Auto-format code with fourmolu

* feat: improve CLI events get and KQL field aliases

- events get: add --at TIMESTAMP flag for direct O(1) lookup via new
  GET /api/v1/events/{id}/time/{ts} endpoint (uses timeseries partition
  key instead of 90d range scan)
- events get: extend default lookback from 24H to 90d so older spans
  resolve without --at
- KQL: map span_name → name, service → resource___service___name,
  trace_id → context___trace_id in WHERE clauses (these are SELECT
  aliases, not real columns; querying them returned HTTP 400)
- CLI: add client-side KQL validation via validateQueryOrDie; bad queries
  now show Megaparsec's precise error instead of an opaque server 400
- queryMetrics: fix error body (was "Invalid signature; \"...\""
  instead of "Invalid query: ...")

* fix: correct synthStackFromSpans service lookup and cleanup

- synthStackFromSpans: replace wrong nested resource["service"]["name"]
  lookup with Telemetry.spanServiceName (uses flat "service.name" key,
  matching the SQL and the existing helper). Service labels were always
  empty before this fix.
- Remove now-unused AesonText import from ApiHandlers
- enrichIssue: add TODO comment noting the N+1 query cost and the
  correct long-term fix (store synthetic stack at ingestion time)
- apiEventGet: move to after apiFacets with proper haddock; fix
  orphaned apiFacets comment that was running into apiEventGet's doc
- Expr.hs: document outputFieldAliases SELECT/WHERE trade-off and the
  exceptionFlattenedFields/flattenedOtelAttributes overlap invariant
- Commands.hs: clarify why validateQueryOrDie is not needed for tail/
  context (those commands have no user-supplied KQL field)

* review: address code-review feedback

- cli: validate --at ISO-8601 timestamp eagerly; an unparseable value now
  prints an error and exits non-zero instead of silently falling back to a
  90d range scan that hides the typo.
- cli: fix Haddock — collapse two stacked '-- |' markers on
  validateEventsOpts into one block so haddock no longer drops the second.
- api: enrichIssue decodes via the typed RuntimeExceptionData record
  instead of pattern-matching raw JSON, so a renamed/removed field is a
  compile error rather than a silent "" default.
- parser: derive exceptionFlattenedFields from flattenedOtelAttributes so
  the "every entry must also live under attributes.exception.* there"
  invariant is enforced automatically.
- api: hoist the empty-spans guard to the top of synthStackFromSpans so
  the sortOn/map allocation is skipped on the empty case; replace
  'maybe "" (\\v -> …)' with foldMap to match codebase style.
- tests: add Web.ApiHandlersSpec covering synthStackFromSpans
  (ordering, error marker, missing service/name/span_id, empty list);
  extend ParserSpec with output-field-alias and exception-COALESCE
  rewriting cases.

* fix(hlint): use sortOn from Relude in synthStackFromSpans

Relude already re-exports sortOn so the explicit List. qualifier is
redundant. The hlint check on the previous commit was already flagging
this; my edit kept the same call shape and inherited the warning.

* docs and readme

* fix: update tests to reflect trace_id alias resolution and resource format

- Update isnotnull(trace_id) test to expect context___trace_id in WHERE
- Fix resource field format in ApiHandlersSpec to use service.name flat key
- Add bracketed service assertion to synthStackFromSpans test
- Simplify span context access and trim verbose comments in ApiHandlers

* fix(telemetry): atMapText falls back to flat dotted keys

getNestedValue now tries the literal joined key before splitting on '.',
so resource maps with flat 'service.name' keys resolve correctly. Also
inlines exceptionFlattenedFields, tightens CLI event-id validation, sets
a 5m wreq timeout for long aggregates, and refactors the span-order
test to compare line indices.

* fix(test): import findIndex in ApiHandlersSpec

* fix(test): replace Data.List.findIndex with list comprehension

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-03 15:06:49 +02:00
Anthony Alaribe
d0350afa52
refactor: CLI agent-mode hardening + cross-cutting LOC reduction (#365)
* refactor: extract mkPageCtx page-handler bootstrap

Replaces 25+ inlined BWConfig + sessionAndProject + ask AuthContext
sequences across Pages/* with a single mkPageCtx helper that returns
(Session, Project, BWConfig) seeded with the always-set fields
(sessM, currProject, config). Net: -102 LoC.

* refactor: extract toastError helper for validation/error responses

Collapses the recurrent 'addErrorToast msg Nothing >> addRespHeaders payload'
shape across Pages/Projects.hs and Pages/Dashboards.hs. Net: -5 LoC plus
+5 LoC for the helper and re-export = breakeven, but eliminates a duplicated
two-line pattern at every site.

* refactor: extract withCardFrame for widget renderer card shell

Collapses the duplicated outer flex-wrapper + optional widget header +
flex-1 fill + surface-raised card shell shared between renderLogsWidget,
renderTraceTable, and renderTable into a single helper.

* refactor: derive FieldTypes/FieldCategoryEnum JSON via WrappedEnumSC

Replaces manual ToJSON/FromJSON instances and the parseFieldTypes,
parseFieldCategoryEnum, fieldTypeToText helper functions with a single
deriving-via clause that piggybacks on the existing WrappedEnumSC wrapper.
Net: -50 LoC.

* refactor: distill CLI; reuse server types (Schema, Paged, MetricsData)

Replace ad-hoc AE.Value field-name probing in the CLI with typed parsing
against the server's existing response types so wire-shape changes
surface as compile errors:

- filterSchema now takes Models.Telemetry.Schema.Schema
- normalizeList decodes via Web.ApiTypes.Paged
- metrics renderers/assertion take Pages.Charts.Types.MetricsData

Also: factor validateDurationOrDie/validateAndNormalizeKind helpers,
fold three case-of-OutputMode triplets through renderWith, derive
ServiceRow/ConfigGetJson/AuthStatusJson/TraceSummary JSON via CustomJSON.

* Auto-format code with fourmolu

* docs: add agentic-pipeline showcase to README

Show the discover -> search -> context -> triage chain with the stable
JSON envelope each command exposes, so a reader scanning the README sees
why the CLI is agent-friendly before getting to the Claude Code Skills
section.

* refactor(cli): apply hlint fixes (errors + warnings + suggestions)

Drops 2 errors and 7 warnings from `hlint cli` to 0:
- Resource.hs: `>> pure n` -> `$> n` for the apply-batch counter
- Commands.hs: `ExceptT . pure` -> `hoistEither`; manual `maybe ... Right` -> `maybeToRight`
- Config.hs: `either (const x) id` -> `fromRight x`
- UI.hs: `T.unlines` -> Relude's `unlines`
- Validate.hs: drop redundant `void`

Plus higher-impact suggestions:
- `data` -> `newtype` for ServicesCommand, ConfigGetOpts, ServicesListOpts, FormEditorName
- `fmap (fmap f) x` -> `f <<$>> x` in 6 list/schema/facets handlers
- `Set.singleton` -> Relude's `one`
- `pure ()` -> `pass` in UI event handlers
- `\_ -> action` -> `const action`
- `(\c -> Map.lookup c idxMap)` -> `(`Map.lookup` idxMap)`

Build green: `All good (8 modules)`. The two remaining hlint hints
(`Data.Set as Set` alias, `universe` import) belong in `.hlint.yaml`
rather than the code.

* fix(test): unwrap IssueL through .base.id in ErrorPatternsSpec

IssueL is a list-view wrapper around Issue (`{ base :: Issue, ... }`),
so the `id` field is reached through `.base`. Four sites in
ErrorPatternsSpec called `issue.id` directly, which type-errored after
IssueL gained its current shape. Matches the pattern used in
src/Pages/Anomalies.hs (`errL.base.id`).

* fix(cli): accept uppercase --since suffixes (1H, 24H)

The platform's TimePicker emits uppercase ("1H", "24H") — the same
strings the CLI itself injects as defaults via buildSearchParams /
runServicesList — but the validator only accepted lowercase, so a user
who copy-pasted the displayed default got rejected client-side. Fold
the suffix to lowercase before checking the whitelist; preserve
original casing in the returned value.

Add a regression test in CLIE2ESpec asserting `--since 1H` is accepted.

* fix(cli): four review nits — flag-aware duration error, single decode, doc/comment fixes

1. validateDurationOrDie now takes the flag name, so --window failures
   read "--window must match …" instead of "--since must match …".
   Three call sites updated (--since x2, --window x1).

2. aggregateServices keeps its `Map.lookup "service"` but with a comment
   pinning the contract: the server aliases resource___service___name to
   the short "service" column in colIdxMap (Pages.LogExplorer.Log.allCols).
   Added an e2e regression in CLIE2ESpec that asserts services-list still
   returns {services, count} so a column rename can't silently zero the
   output.

3. docs/cli.md agentic-incident-workflow: replace .items[] with .data[]
   in the issues-list and log-patterns-list jq pipes — they'd break
   against the now-normalised list envelope.

4. runEventsContext --summary used to walk logsData twice (once in
   normalizeEventsResponse, once in withTraceSummary). Introduce a
   `DecodedEvents` record that carries idxMap/rows/raw, decode once at
   the call site, and thread it through both functions.

* perf(cli): back services list with /api/v1/facets instead of 10k-event scan

Previously runServicesList fetched up to 10,000 events and aggregated
client-side because /api/v1/metrics returned empty for `summarize ... by`.
The facets endpoint (precomputed in apis.facet_summaries via a background
job) is one indexed lookup — sub-millisecond regardless of project size.

- Drop the events-scan fallback; switch to GET /api/v1/facets?field=resource.service.name
- ServiceRow loses the `last_seen` field (facets don't track per-value
  recency). The remaining {name, events} is what the use-case actually
  needs ("what services exist? how busy are they?").
- e2e regression in CLIE2ESpec already covers the {services, count}
  envelope; comment updated to pin the facets-backed contract.
- docs/cli.md and monoscope-skills/README.md output-shape tables drop
  last_seen.

* chore: trailing review nits — unused asks, V.take, normaliseList warning, debug-comment

Server side:
- dashboardsGetH: drop the appCtx ask; bw.config.liveReloadDashboards
  carries the flag now that mkPageCtx threads config through.
- teamAlertsGetH: drop appCtx + sess + project, all genuinely unused
  bindings.

(dashboardGetH / dashboardTabGetH / dashboardTabContentGetH /
unifiedMonitorOverviewH still bind appCtx — they read appCtx.env.* which
isn't on bw.config, so the binding is required.)

CLI:
- takeFirstEvent: V.take 1 arr instead of take 1 (toList arr); avoids the
  Vector→List→Vector round-trip.
- Resource.normalizeList split into normalizeListE returning Either, with
  Left carrying the raw payload on shape mismatch. runList now flags the
  unrecognised case via printDebug so contributors see when the server
  envelope drifts.
- Core.renderParams: comment pinning the "debug-only, NOT URL-encoded,
  redact if you ever pass auth via query params" contract.

* fix(cli): CI doc uses .data, emitFirstEventId avoids list round-trip

- docs/cli.md CI/Automation example said `jq '.items | length'` —
  inconsistent with the agentic-workflow section above and broken since
  normalizeList moved everything to `.data`.
- emitFirstEventId now uses `arr V.!? 0` instead of `head (toList arr)`
  — same V.take optimisation as takeFirstEvent, no intermediate list
  allocation.

Bug #2 from the review (renderEventsTable in runEventsContext receiving
raw `val`) is intentional: the table renderer reads `cols`/`logsData`
out of the raw envelope to produce a dense table; the JSON path uses
the named-field `enriched` form. Passing `enriched` to the table
renderer would render an empty table because the normalised shape
doesn't carry `cols`/`logsData`. The split is by design — table mode is
column-oriented, JSON mode is record-oriented.

* fix(cli): six review items — agent-mode truthy check, --first table parity, single decode, derived facets, forall pinning

3. isAgentMode no longer fires on `CI=` (empty) or `CI=false` — many dev
   shells export CI unconditionally; treat empty/"false"/"False"/"FALSE"/"0"
   as off.
4. --first parity in table mode: takeFirstRow trims the raw envelope's
   logsData to one row, so table output matches the JSON path's
   takeFirstEvent slice instead of silently dumping all rows.
5. Free-floating Haddock above DecodedEvents merged into the actual
   docstring (the orphan would have triggered a -Wmissing-signatures-ish
   warning eventually).
6. runEventsSearch now decodes once and threads DecodedEvents through
   normalizeDecoded, matching the pattern in runEventsContext. No more
   double walk of logsData on the JSON path.
7. parseServiceFacets decodes via Models.Apis.Fields.FacetValue
   (FromJSON via DAE.CustomJSON) — drops 6 lines of hand-rolled
   KM.lookup/AE.String/AE.Number unpacking and surfaces server wire
   changes at compile time.
8. withMetricsData now forwards aeson's Error message through
   printDebug so a server envelope drift is visible under --debug
   instead of vanishing into the raw-JSON fallback.

Plus: explicit `forall es a b` on apiGetJson/apiPostJson/apiPutJson/
apiPatchJson so TypeApplications at call sites
(`apiPostJson @_ @_ @AE.Value`) survive future constraint reordering by
fourmolu/hlint. Without this, alphabetical resort of
`(AE.FromJSON b, AE.ToJSON a, Environment :> es, ...)` flips the
implicit type-var order and the third `@AE.Value` lands on `es` instead
of the body type.

Run fourmolu over the touched files; #10 (double blank lines) is
auto-fixed.

* chore: round-three review nits — null instead of "", helpCases coverage, dead code

3. normalizeDecoded now emits `null` for missing fields (row shorter
   than colIdxMap) instead of silently coercing to "". Agents can now
   tell "field present but empty" from "field missing".
4. Comment on the --debug setEnv site explaining the single-shot, pre-
   thread-spawn placement (so future readers know it's not a race).
6. emitCompletion lifts to IO once at the entry instead of per-branch.
7. CLIBinarySpec helpCases extended with `facets`, `schema`, `services
   list`, `events {search,context,tail}`, `metrics {query,chart}`,
   `auth login`, `completion` — every top-level subcommand now has a
   --help smoke test.
8. dashboardGetH/dashboardTabGetH: replace appCtx.config.hostUrl with
   bw.config.hostUrl now that mkPageCtx threads config through; appCtx
   binding stays because appCtx.env.apiKeyEncryptionSecretKey isn't on
   bw. dashboardTabContentGetH untouched — no mkPageCtx, no bw.

Style:
- Drop dead extractNumericRows from Pkg.CLIFormat (no callers since the
  metrics path moved to MetricsData).
- Export validateDurationFor so the doctest examples (which reference it)
  resolve at doctest runtime.

Re #2 (parseServiceFacets via FromJSON) and #10 (double blank lines):
already landed in commit 2a54bd0b.

Re #5 (renderParams allowlist): keeping the comment-only enforcement.
The CLI doesn't put credentials in query params today (Bearer + project
go via headers), and a type-level distinction would touch every API
helper for a hypothetical leak. The doc note is sufficient guard for
contributor review.

* fix(cli): trace-summary error count, KQL injection guard, small wins

Core bug:
- withTraceSummary used `cell "severity"` to count errors per trace, but
  the events colIdxMap (Pages.LogExplorer.Log.allCols) doesn't expose a
  "severity" column — only the boolean "errors" flag. Result: every
  trace summary reported error_count=0 regardless of traffic. Now reads
  cell "errors" and treats "true" as 1. Also added a defensive fallback
  for service ("service" <|> "resource.service.name") in case a future
  column-set rename hides the alias.

KQL safety:
- runEventsGet escapes `\` and `"` in opts.eventId before splicing into
  the query string. Event IDs aren't UUIDs (synthetic-… span ids,
  log_pattern integers, …) so format validation is wrong; escaping
  defends against injection regardless of shape.

Small wins:
- isAgentMode: fold suffix to lowercase once via `map toLower v` rather
  than enumerating "false"/"False"/"FALSE".
- filterSchema: `Map.take n filtered` instead of
  `Map.fromList (take n (Map.toList filtered))` (containers ≥ 0.6.3).

Doctests note:
- The earlier "doctest comment" the review flagged was already removed
  in a prior commit. Pure validators are still covered end-to-end via
  CLIE2ESpec ("--since 1xyz", "--kind banana") which exercise the real
  binary. Wiring CLI/* into the lib doctest target would need moving
  the modules into src/ — out of scope for this round.

Pushed back:
- AuthStatus method ADT: the strings "env"/"token" appear in two
  adjacent lines of one function. A two-constructor ADT + custom ToJSON
  is more boilerplate than the leak it prevents (no other consumer).
- Map merge tuple readability: a record would cost two lines and one
  data declaration to save zero compile errors. The merge fits on one
  line; leaving it.

* chore: clarify intent — `mempty` empty bodies, capFacets exhaustiveness comment

- Pages/Projects.hs: replace seven `toastError "…" ""` and three
  `addRespHeaders ""` with `mempty`. Reads as "no body" rather than
  "empty string body" — the underlying type is `Html ()` whose Monoid
  identity is the empty Html, same wire effect.
- cli/Main.hs `capFacets`: comment on the trailing wildcard clause so
  the (Just n) + non-Object fallthrough is explicitly documented vs the
  Nothing case the first clause already handles.

Re facets in CLIBinarySpec helpCases: already added in 76ebf71c.

* chore: clear hlint warnings across src/

BackgroundJobs.hs: 6× displayException + 2× fromException were
resolving to Control.Exception.* (because the file did
`import Control.Exception (ErrorCall (..))` which exposed the
Control.Exception namespace to GHC's resolver and tripped
.hlint.yaml's "use Relude version" rule). Drop the unqualified
import; ErrorCall now goes through `Control.Exception qualified as CE`
so the only `Control.Exception` reference is the explicit qualifier
and unqualified displayException/fromException resolve through Relude.

Pages.Anomalies.hs:
- NE.head -> head (Relude)
- two whenJust lambdas -> point-free (toHtml composition)
- when (... `notElem` ...) -> unless (... `elem` ...)

Pages.Settings.hs: sortOn fst -> sortWith fst (no caching needed for
projection on first component).

Pages.Telemetry.hs: MapS.singleton k v -> one (k, v) (Relude polymorphic
container constructor).

Pages.LogExplorer.Log.hs: two `\j -> (j, x)` lambdas -> tuple sections
(TupleSections is part of GHC2024, already on).

Verified clean: `hlint src/BackgroundJobs.hs src/Pages/Anomalies.hs
src/Pages/Settings.hs src/Pages/Telemetry.hs src/Pages/LogExplorer/Log.hs`
reports "No hints".

* Auto-format code with fourmolu

* fix(cli): forward Paged decode error, lenient body decode, unambiguous null

3. normalizeListE now returns Either (rawValue, reason) — when an Object
   fails @Paged@ decoding the aeson Error message is captured and
   surfaced through runList's --debug log. Contributors see *what*
   shape changed instead of a generic "did not match envelope".
   normalizeList stays pure with `either fst id . normalizeListE`.

4. httpExToError now uses TE.decodeUtf8With lenientDecode for both the
   status reason and the response body. The body field of
   StatusCodeException is the truncated raw bytes (≤1024B) and may be
   gzip leftovers, HTML, or arbitrary upstream junk; strict decodeUtf8
   would throw at the error-handling boundary, masking the real HTTP
   failure with a UnicodeException.

6. isAgentMode: replace `not (null v)` with `v /= ""` to dodge the
   Prelude.null vs Relude container-null ambiguity. v is String here.

Re prior style notes:
- Double blank lines between top-level bindings: produced by fourmolu
  format pass; project convention is enforced by the formatter.
- AuthStatusJson / ConfigGetJson: kept as records — the CamelToSnake
  derivation pins the wire shape (an inline AE.object [...] silently
  drifts when fields are added).
- runConfigGet OutputMode asymmetry vs init/set: noted; init takes no
  output path (interactive form) and set has no payload to format.
  Follow-up if it surfaces in user feedback.
- normalizeList = either fst id . normalizeListE (was either id id;
  shape changed when Left started carrying the reason).
- CLI/UI.hs `currentValue = T.strip . unlines . BE.getEditContents`:
  verified — `unlines` here is the Relude re-export, which is polymorphic
  over the IsList/Foldable instance. Compiles clean.

* fix(test): unwrap IssueL through .base.issueType in AnomaliesSpec

Same pattern as the ErrorPatternsSpec fix in f0ab87d7: IssueL wraps
Issue in `base`, so the issueType field lives at `c.base.issueType`,
not `c.issueType`.

* fix(test): add http-types to integration-tests build-depends

CLIE2ESpec imports Network.HTTP.Types.Status (statusCode) for the
reachability probe but http-types wasn't on the test target's deps.
The library has it (line 173), the test suite didn't inherit it.

* fix(test): drop System.Environment lookupEnv import — Relude already exports it

* fix(cli/test): try @SomeException → tryAny, plus seven review nits

Build break: UnliftIO.Exception.try's forall is (m, e, a); the @SomeException
TypeApplication landed on m and tripped a kind error. Switch to tryAny
(MonadUnliftIO m => m a -> m (Either SomeException a)) — same semantics,
no type-arg juggling. Drops the redundant Data.List (isInfixOf) import
along the way (Relude already has it).

Review items:
- AuthStatus: extract printStatus helper for the env/token branches —
  3 lines repeated → 1 helper.
- apiFacets: replace addUTCTime (-86400) with negate nominalDay (Data.Time
  re-export). No more 86400-second magic.
- capFacets: drop the now-redundant `Nothing` clause; the wildcard at the
  end already covers it (and `Just n` + non-Object).
- parseDurationMs: T.toLower the input via a view pattern so --watch 5M
  is treated like --watch 5m, matching validateDurationFor's case-insensitive
  suffix check.
- reqOpts: collapse the broken multi-line `& W.header "Accept" .~ [...]`
  formatter artifact into a single parenthesised line so subsequent `&`
  chain elements align under W.defaults.
- Resource.runList: route the unrecognised-envelope reason through
  printError (always-on stderr) instead of printDebug — agents
  expecting {data, pagination} need to see when the server returns
  something else.
- DecodedEvents: decode count / has_more / cursor at construction
  instead of carrying the raw KeyMap. Three lookups eliminated, intent
  is local to decodeEvents.

Re minor nits noted:
- isAgentMode/isDebugMode re-read env on each call: kept; called at
  most a handful of times per invocation, no hot path.
- Data.Char (toLower): kept the explicit import — Relude does re-export
  it but the explicit form documents which `toLower` we mean (vs
  Data.Text.toLower which operates on Text).

* fix(test): restore Data.List (isInfixOf) — Relude does not re-export it

* fix(test+ci): two CLI test failures, default e2e suite to the public demo project

CI failures:

1. CLIBinarySpec "auth status exits and shows status message" — when CI
   is set, isAgentMode auto-enables and `auth status` emits the JSON
   envelope ({authenticated, method, api_url, project}). The test
   predicate only matched plain-text "Not authenticated" / "Authenticated".
   Extend it to also accept the JSON shape ("authenticated" key).

2. CLIBinarySpec "completion --help" exited 1. The completion subparser
   declared a positional SHELL argument but didn't attach `<**> helper`,
   so `--help` got consumed as the SHELL value and fell through to the
   "unknown shell" branch. Adding helper makes optparse intercept --help
   and emit usage with exit 0.

Default e2e suite to prod demo project:

CLIE2ESpec.getE2EConfig now defaults @MONOSCOPE_API_URL@ to
https://api.monoscope.tech and @MONOSCOPE_PROJECT@ to the public demo
00000000-0000-0000-0000-000000000000. Only @MONOSCOPE_API_KEY@ has no
safe default (read keys for other projects would leak). The pendingWith
message names the missing variable specifically so it's obvious how to
unlock the suite.

Wired up in CI: pullrequest.yml's integration-tests step now reads
MONOSCOPE_API_KEY from secrets.MONOSCOPE_DEMO_API_KEY. Repo owner needs
to mint a read-only key on the demo project and store it in repo
secrets — then every PR/push runs the suite against prod. Forks (no
secret) still get the CLIBinarySpec smoke tests.

docs/cli.md updated with the new default-driven workflow.

* fix(test): assert specific merge edge instead of fragile global count

ErrorPatternsSpec test 12 ("Full LLM pipeline") asserted
`length mergedTypeErrors == 1` after merging one (childId, canonId)
pair. The query counted ALL TypeError patterns with canonical_id set
across the project, so any earlier test that merged a TypeError made
the count drift up. CI now sees 2 because some upstream change in the
LLM/notify pipeline produced an additional pre-merged pattern by the
time test 12 runs.

Switched to a behaviour-level assertion: query the specific childId's
canonical_id and require it equals canonId. This tests what
`assignErrorsToCanonical` actually does, with no dependence on the
global state earlier tests left behind.

* refactor(cli): lens-based shape ops, guards, precomputed indices

- takeFirstEvent / takeFirstRow rewritten with Data.Aeson.Lens
  (`AL.key … . AL._Array %~ V.take 1`). 14 lines of pattern-match-and-
  reinsert collapsed to two one-liners. AL is qualified to avoid
  collision with the `key` record fields on ConfigSet/GetOpts.
- foldFiltersIntoQuery: case (T.null …, T.null …) → pattern guards.
  Reads top-down instead of as a tuple-match puzzle.
- normalizeListE: AEKM.member → has (AL.key …) lens accessor; the
  branch matches on the value directly, no need to destructure
  AE.Object obj first.
- extractColIdxMap: round → floor on the Scientific. Column indices are
  already-integral non-negative ints; floor documents that intent and
  drops banker's rounding semantics that don't apply here.
- parseDurationMs: drop the bare-number-as-seconds fallback. validateDurationFor
  rejects unsuffixed values upstream, so the only paths that reach
  parseDurationMs have a recognised suffix; a typo now returns the
  5s default loudly instead of silently treating "5" as five seconds.
- runMetricsChart: validateDurationOrDie "--watch" before entering the
  forever loop, so a bad --watch string fails at flag parse rather
  than burning HTTP requests every 5s.
- withTraceSummary: precompute svcIdx / errIdx / tidIdx once instead
  of re-Map.lookup'ing the same column path per row. Saves one Map
  lookup per cell per row — measurable on large
  `events context --summary` payloads.

Build deps: cli exe gains `lens-aeson` (already in lib deps for the
server's Data.Aeson.Lens usage). hpack regenerated monoscope.cabal.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-05-02 23:38:22 +02:00
Anthony Alaribe
497ba79a7e fix: add /api/v1 prefix to all CLI resource paths, extend test routing
All CLI resource paths were missing the /api/v1 prefix, causing 404s
(e.g. GET https://api.monoscope.tech/issues instead of /api/v1/issues).

- cli/CLI/Resource.hs: resourcePath now returns /api/v1/<slug>
- cli/Main.hs: all explicit paths (/issues, /me, /project, /share,
  /schema, /api_keys, dashboard widgets) prefixed with /api/v1
- src/Pkg/TestUtils.hs: extend runHTTPtoServant to route GET/POST for
  all api/v1 paths (issues, endpoints, log_patterns, me, project) to
  real handlers; support POST/PUT/PATCH/DELETE dispatch
- test/integration/CLI/CLISpec.hs: add cliPost helper and CLI resource
  command tests that exercise /api/v1/* paths through the HTTP router
- test/integration/Web/ApiV1Spec.hs: document path list as CLI contract
2026-05-02 01:02:29 +02:00
Anthony Alaribe
f0c5653a19 fix: test correctness, doctest package imports, channel gating, distillation
- Two-pass spike detection: run detectLogPatternSpikes twice in tests
  (first pass pends, second pass fires) to match the algorithm
- Doctest imports: qualify with package name (-XPackageImports) and
  filter --color/--jobs/--match flags unsupported by doctest runner
- Channel gating: add countingSendGated in Settings for slack/discord/
  whatsapp/pagerduty; wire channel setup in BotTestHelpers; clear
  disabled_channels in test teardown
- AnomaliesSpec: use frozenTime instead of getCurrentTime for determinism;
  add missing summary column to INSERT
- LogPatternsSpec embedding test: isolate pre-existing patterns via
  merge_override before running patternEmbeddingAndMerge
- ErrorPatternsSpec: exclude PatternEmbeddingAndMerge from background
  jobs run to allow explicit embedding control in test 12
- BusinessFlowsSpec: add ON CONFLICT DO UPDATE for idempotent test setup
- Fix Slack report assertion: T.toTitle transforms "TEST" → "Test",
  so assert "🧪 Test:" instead of the original "🧪 TEST"
- BackgroundJobs SQL: cast cooldownHours to ::int in make_interval call
- PatternMerge: add ORDER BY id to getUnembeddedLogPatterns for stable
  ordering under the 500-row limit
- Distill Settings.hs: merge countingSend/countingSendGated into one
  helper with Maybe Text gate argument
- Distill TestUtils.runAsBase: replace case result with >>= either
- Distill CLIBinarySpec: inline single-use combined let binding
- Add golden files for Slack conversations API responses
2026-05-02 00:42:30 +02:00
Anthony Alaribe
78aa5ea8a0 feat(api,cli): expand public API surface with project, endpoints, log patterns, issues, teams and members
Adds CLI subcommands and JSON handlers covering /me, /project,
/endpoints, /log_patterns, /issues, /teams and /members, plus shared
paginated envelope and generic bulk result parameterised by id type.
Extracts share-event persistence into Models.Apis.ShareEvents and
introduces a SnakeJSON type alias to collapse repeated deriving-aeson
boilerplate.
2026-04-22 12:58:09 +02:00
Anthony Alaribe
f61211f950 refactor(api,cli): address PR review feedback
- CLI error rendering via renderAPIError (HTTP code + body), drops raw Show output
- decodeBody no longer coerces empty body to "null"; surfaces explicit error
- httpExToError includes URL/method context across all branches
- runApplyResource counts per-file failures; exits non-zero only at the end
- Collapse redundant CLI wrappers; add withCfgMode, resourceIdPath helpers
- readYamlOrJson branches on .json extension for clearer parser errors
- Rename apiPost to apiPostUnauth to flag its device-code bootstrap role
- BulkFailure serialises as {id, error}; add Eq for test assertions
- Integration tests for bulk failure shape, unknown action 400, monitor 404
- CLI help-case test asserts invoked subcommand name appears in output
- Add --version subcommand test
- docs/cli.md: fix apply wording, drop anomalies bulk row, sync 48h TTL marker
2026-04-15 22:39:19 +02:00
Anthony Alaribe
a5633cda89 refactor(api,cli): PR review cleanup pass
- Fix anomalies bulk UUID encoding + archive via new archiveAnomaliesAndIssues helper
- Fix mkPaged hasMore off-by-one; hide Paged constructor behind mkPaged
- Fix endpointsListH per_page count via Endpoints.countEndpointsByProject
- Collapse DashboardFull to nest DashboardSummary; drop mkSummary HasField shim
- Collapse monitor/dashboard/api-key lifecycle handlers with withRefetch helper
- Drop unused monitor list filter/since query params
- Scope Resource.runApply to dashboards (removes dead MonApply path)
- Add dashboards unstar CLI command; fix star help text to drop toggle claim
- Extract CLI runAPI + export withResult helpers to dedupe Main.hs dispatch
- Fix CLI decodeBody to surface parse errors on empty body instead of silent null
- Replace runStaticUUIDRef crash on empty pool with UUID.nextRandom fallback
- Tighten CLI jsonCall constraint to IOE only (HTTP travels via action param)
2026-04-15 20:51:25 +02:00
Anthony Alaribe
f8526d866a feat(api): REST API v1 + CLI
Public JSON REST API under /api/v1 for monitors, dashboards, api-keys,
events, metrics, anomalies, issues, log-patterns, endpoints, projects,
share links, and `me` — mirrors HTML handler behaviour but runs in
ATBaseCtx with API-key auth.

Handlers & types (Web.ApiHandlers, Web.ApiTypes):
- Thin JSON adapters over existing Models.*. A single `bulkExec`
  dispatcher backs monitor/dashboard/log-pattern/anomaly bulk endpoints;
  anomalies bulk routes through Anomalies.acknowlegeCascade so API
  cascade semantics match the HTML path.
- Issues share a `mkSummary` HasField-polymorphic projection used by
  both IssueSummary and IssueFull, and an IssueOp ADT that compresses
  ack/unack/archive/unarchive into one runner.
- Paged envelope (items/total/page/per_page/has_more) + mkPaged.
- Log patterns and anomalies are thin aliases over issues filtered by
  issue_type so one storage path backs three UI views.

Shared code consolidated:
- Monitors created via API parse KQL → SQL via Pkg.Parser, matching the
  form path (previously logQueryAsSql was empty).
- ProjectApiKeys.encodeApiKeyB64 consolidates encrypt+base64 previously
  duplicated across ApiHandlers, Pages.Settings, Pages.Projects.
- Dashboards.getDashboardByFilePath replaces an inline raw-SQL query in
  the YAML-apply handler.
- Pkg.DeriveUtils.JsonValueSchema DerivingVia wrapper replaces five
  manual ToSchema fallbacks.
- Data.Effectful.UUID.runStaticUUIDRef: shared IORef-backed UUID pool so
  deterministic test UUIDs persist across handler invocations in a
  single test. Pkg.TestUtils exposes runAsBase / trUUIDRef.
- Models.Apis.Issues.IssueType/IssueEvent now derive Bounded/Enum and
  gain ToSchema + FromHttpApiData through WrappedEnumSC, flowing
  through Servant capture/body params + OpenAPI without manual
  instances. Models.Apis.Endpoints.Endpoint derives HI.DecodeRow.

CLI (cli/):
- Top-level commands wired against the new endpoints.
- Shared OutputMode (JSON/YAML/Table) + detectOutputMode + renderByMode
  + apiGet/apiGetJson/apiPost/apiPostJson/apiPutJson/apiPatchJson/
  apiDelete helpers in CLI.Core.
- package.yaml for the `monoscope` exe: Paths_monoscope added to
  generated-other-modules (needed for --version); base listed as an
  explicit dependency; `Relude as Prelude` mixin dropped so the autogen
  Paths_monoscope.hs `import Prelude` is unambiguous. All cli modules
  already import Relude explicitly. Fixes a CI build failure on Linux.

Integration coverage: 18 tests in Web.ApiV1Spec covering OpenAPI spec,
auth, schema/monitors/events/metrics endpoints, monitor and dashboard
CRUD + lifecycle, API key CRUD, events body query, share links.
2026-04-15 20:51:25 +02:00