monoscope/test/integration
History 2026-11-16 03:47:38 +00:00
..
CLI Consolidate src/Pkg (45 -> 29 modules); nested monitor YAML config 2026-11-05 00:54:10 +01:00
Data/Effectful Read-only dashboards: one refusal on every route, linking refuses built-ins, template tags and starring (#844) 2026-11-10 22:45:36 +01:00
Opentelemetry Traces: per-step tokens and cost in the waterfall, LLM cost in the trace header 2026-11-16 03:47:38 +00:00
Pages Heatmap drill-down resolves request_kql before opening the Explorer (#916) 2026-11-16 02:43:14 +01:00
Pkg Telemetry and infrastructure specs: self-contained lifecycles (#778) 2026-11-02 14:03:23 +01:00
System Unify autonomous agents across Slack, web and native jobs (#619) 2026-11-02 11:15:00 +01:00
Web Floating AI assistant that acts on, navigates and reloads the UI (#912) 2026-11-15 09:10:03 +01:00
BackgroundJobsSpec.hs Reports become AI routines; routines and wide tables are easier to read (#882) 2026-11-12 22:51:43 +01:00
BillingSpec.hs Reports become AI routines; routines and wide tables are easier to read (#882) 2026-11-12 22:51:43 +01:00
EndpointsSpec.hs Generate application links from Servant routes and simplify Kafka consumer state (#799) 2026-11-06 18:51:14 +01:00
ErrorFingerprintStabilitySpec.hs Consolidate shared UI components, distill backend, enforce per-handler project membership (#670) 2026-10-11 09:48:06 +02:00
EvalConformanceSpec.hs Read-only dashboards: one refusal on every route, linking refuses built-ins, template tags and starring (#844) 2026-11-10 22:45:36 +01:00
Main.hs Billing and background-job specs: self-contained lifecycles (#775) 2026-11-02 12:07:04 +01:00
MinioManagerSpec.hs fix: reuse HTTP transport for replay object storage (#554) 2026-09-08 12:37:24 +02:00
MonitoringSpec.hs Reports become AI routines; routines and wide tables are easier to read (#882) 2026-11-12 22:51:43 +01:00
README.md Consolidate src/Pkg (45 -> 29 modules); nested monitor YAML config 2026-11-05 00:54:10 +01:00
ReplaySpec.hs Tests use no Kafka broker; the shared producer refuses empty brokers (#801) 2026-11-06 15:43:06 +01:00
Spec.hs Unify team email resolution so @everyone test notifications reach all members 2026-02-15 20:28:48 +01:00
SpecHook.hs Repository specs: absorb GitSyncSpec and Pkg.GitSpec into CodeContextSpec (#789) 2026-11-02 23:55:09 +01:00

Integration Testing Guide

Running Tests

Standard CI Mode (uses golden files only)

USE_EXTERNAL_DB=true cabal test integration-tests

Tests will use cached golden files and fail if any are missing.

Updating Golden Files (makes real API calls)

UPDATE_GOLDEN=true USE_EXTERNAL_DB=true cabal test integration-tests

This will:

  • Make real HTTP calls to Twilio, Discord, Slack, etc.
  • Make real LLM calls to OpenAI API
  • Cache all responses to golden files
  • Requires credentials in .env file

Running Specific Test Suites

USE_EXTERNAL_DB=true cabal test integration-tests \
  --test-option="--match=/Discord Bot/"

Golden File System

Golden files cache real API responses to make tests:

  • Fast: No network calls in CI
  • Deterministic: Same response every time
  • Runnable without credentials: Golden files checked into repo

When to Update Golden Files

  1. After changing bot response format - Regenerate bot golden files
  2. After changing LLM prompts - Regenerate LLM golden files
  3. After external API changes - Regenerate HTTP golden files
  4. When adding new tests - Generate golden files for new test cases

Golden File Locations

  • HTTP responses: test/golden/*.json (URL-based filenames)
  • Bot responses: test/golden/bots/{discord,slack,whatsapp}/*.json
  • LLM responses: test/golden/agentic_*.json (hash-based filenames)

Missing Golden Files

If you see errors like:

Golden file not found: test/golden/https___api.twilio.com_...json
Run tests with UPDATE_GOLDEN=true to create it

This means a new test was added without golden files. To fix:

UPDATE_GOLDEN=true USE_EXTERNAL_DB=true cabal test integration-tests

Important: Commit the new golden files to the repository!

External Database Mode

Integration tests use USE_EXTERNAL_DB=true for:

  • Better isolation (each test run gets fresh DB from template)
  • Faster test runs (template cached, no migration reruns)
  • Reliable cleanup (force-terminate connections)

Requirements

  • PostgreSQL running on localhost:5432
  • User: postgres, Password: postgres (or set EXTERNAL_DB_URL)
  • TimescaleDB extension installed

Database Lifecycle

  1. Template database created: monoscope_test_template
  2. Test database cloned: monoscope_test_{uuid}
  3. Tests run with isolated database
  4. Connections terminated and database dropped

Test Credentials

Tests need real credentials in .env for UPDATE_GOLDEN=true mode:

TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
DISCORD_BOT_TOKEN=...
DISCORD_PUBLIC_KEY=...
SLACK_BOT_TOKEN=xoxb-...
OPENAI_API_KEY=sk-...

Copy configs from .env.prod (with production credentials).

Note: Golden files are designed as golden tests - they capture real provider responses once, then replay them for fast CI runs without credentials.

Troubleshooting

"database is being accessed by other users"

  • Connection cleanup improved - should not occur anymore
  • If still occurring, check for long-running background jobs

"relation telemetry.otel_logs_and_spans does not exist"

  • Table is in public schema, not telemetry schema - should be fixed

Discord signature verification failures

  • Verify DISCORD_PUBLIC_KEY in .env matches test fixtures
  • Check timestamp handling in test requests

Golden file response mismatches

  • Run UPDATE_GOLDEN=true to regenerate from live APIs
  • Compare diffs to understand what changed
  • Update test expectations if API behavior changed legitimately