monoscope/docs/anomaly-detection-system.md
2025-09-14 19:55:26 +02:00

6.7 KiB

Anomaly Detection System Documentation

Overview

The Monoscope anomaly detection system automatically identifies and tracks changes in API behavior, including:

  • New endpoints, request/response shapes, and field formats
  • Runtime exceptions and errors
  • Performance degradations and threshold violations

The system is designed to minimize noise by grouping related anomalies into unified issues that developers can easily understand and act upon.

Architecture

1. Data Flow

HTTP Requests → PubSub/OTLP → Span Processing → Entity Extraction → Database Triggers → Background Jobs → Issue Creation → Notifications

2. Key Components

A. Request Ingestion (src/ProcessMessage.hs)

  • Receives HTTP request data via PubSub messages or OTLP protocol
  • Converts to OpenTelemetry span format for unified processing
  • Stores in otel_logs_and_spans table with computed hashes

B. Entity Extraction (src/ProcessMessage.hs:processSpanToEntities)

  • Extracts from each span:
    • Endpoints: Unique API routes (method + path + host)
    • Shapes: Request/response structure for each status code
    • Fields: Individual data fields with their paths
    • Formats: Field value patterns and examples
  • Uses XXHash for deterministic hash generation
  • Implements caching to reduce database operations

C. Anomaly Detection (Database Layer)

  • PostgreSQL triggers on entity tables:
    • endpoints_created_anomaly - Fires on new endpoint insertion
    • shapes_created_anomaly - Fires on new shape insertion
    • fields_created_anomaly - Fires on new field insertion
    • format_created_anomaly - Fires on new format insertion
    • error_created_anomaly - Fires on new error insertion
  • Stored procedure apis.new_anomaly_proc():
    • Creates anomaly records in apis.anomalies table
    • Batches anomalies into background jobs by project/type
    • Deduplicates to prevent multiple jobs for same anomalies

D. Background Job Processing (src/BackgroundJobs.hs)

  • FiveMinuteSpanProcessing: Processes spans to extract entities
  • OneMinuteErrorProcessing: Extracts errors from span events
  • NewAnomaly: Converts anomalies to user-facing issues

E. Issue Management (src/Models/Apis/Issues.hs)

  • Three issue types:
    • APIChange: Groups endpoint/shape/format changes by endpoint
    • RuntimeException: Individual issues for each error pattern
    • QueryAlert: Threshold violations from monitoring
  • Issue lifecycle: Created → Enhanced (LLM) → Acknowledged → Archived

F. Notification System (src/Pkg/Mail.hs)

  • Multi-channel support: Email (Postmark), Slack, Discord, WhatsApp
  • Template-based notifications with project-specific preferences
  • Batched notifications to prevent spam

Implementation Details

Hash Calculation

Hashes are hierarchical to enable efficient lookups:

  • Endpoint: XXHash(projectId + host + method + urlPath)
  • Shape: XXHash(endpointHash + statusCode + sortedFieldPaths)
  • Field: XXHash(endpointHash + category + fieldPath)
  • Format: XXHash(fieldHash + valuePattern)

Anomaly Types

  1. Endpoint Anomalies

    • Triggered when a new API endpoint is discovered
    • Groups all related changes under single issue
  2. Shape Anomalies

    • Triggered when request/response structure changes
    • Tracks added/removed/modified fields
  3. Format Anomalies

    • Triggered when field value patterns change
    • Useful for detecting breaking changes in data formats
  4. Runtime Exceptions

    • Extracted from OpenTelemetry span events
    • Groups by error type, message, and stack trace

Error Detection Process

  1. Every minute, queries spans with error indicators:

    • Status code = 'ERROR' or '2'
    • Events containing 'exception' or 'error'
    • Attributes indicating errors
  2. Extracts error details from span events:

    • Exception type, message, stack trace
    • HTTP context (method, path)
    • Service and technology info
  3. Creates runtime exception issues with:

    • Error grouping by hash
    • Occurrence counting
    • First/last seen timestamps

Issue Enhancement

  • Optional LLM integration for better issue titles and recommendations
  • Analyzes code changes to suggest migration complexity
  • Provides actionable recommendations for developers

Database Schema

Key Tables

  • apis.anomalies - Raw anomaly records (legacy, being phased out)
  • apis.issues - User-facing issues with enhanced information
  • apis.endpoints - Discovered API endpoints
  • apis.shapes - Request/response structures
  • apis.fields - Individual data fields
  • apis.formats - Field value patterns
  • apis.errors - Runtime exceptions

Important Columns

  • hash - Deterministic identifier for deduplication
  • endpoint_hash - Links shapes/fields to endpoints
  • issue_data - Polymorphic JSON for different issue types
  • llm_enhanced_at - Tracks AI enhancement status

Configuration

Project Settings

  • redact_fields_list - Fields to redact from stored data
  • notification_channels - Enabled notification methods
  • payment_plan - Affects rate limiting and features

Environment Variables

  • Rate limits for free tier projects
  • OpenAI API key for enhancement
  • Notification service credentials

Testing Considerations

Current Test Coverage

  • ProcessMessageSpec - Basic message processing
  • AnomalyListSpec - UI rendering (currently disabled)
  • Integration tests for entity extraction
  1. Trigger execution validation
  2. Error extraction edge cases
  3. Notification delivery confirmation
  4. Issue grouping logic
  5. Hash collision handling

Common Operations

Monitoring Anomaly Detection

-- Check recent anomalies
SELECT * FROM apis.anomalies 
WHERE created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC;

-- Check background job queue
SELECT * FROM background_jobs 
WHERE tag = 'NewAnomaly' 
AND status = 'pending';

Troubleshooting

  1. Missing Anomalies

    • Check if triggers are enabled
    • Verify background jobs are running
    • Check for rate limiting on free tier
  2. Duplicate Issues

    • Verify endpoint hash calculation
    • Check unique constraints on issues table
  3. Missing Notifications

    • Verify notification channels configured
    • Check notification service logs
    • Verify user email addresses

Future Improvements

  1. Performance

    • Implement bloom filters for hash lookups
    • Optimize shape comparison algorithms
    • Add caching for frequently accessed data
  2. Features

    • Machine learning for anomaly severity scoring
    • Custom anomaly detection rules
    • Webhook integrations for notifications
  3. Reliability

    • Add retry mechanism for failed notifications
    • Implement dead letter queue for failed jobs
    • Add health checks for detection pipeline