mirror of
https://github.com/monoscope-tech/monoscope.git
synced 2026-11-16 10:18:01 +00:00
6.7 KiB
6.7 KiB
Anomaly Detection System Documentation
Overview
The Monoscope anomaly detection system automatically identifies and tracks changes in API behavior, including:
- New endpoints, request/response shapes, and field formats
- Runtime exceptions and errors
- Performance degradations and threshold violations
The system is designed to minimize noise by grouping related anomalies into unified issues that developers can easily understand and act upon.
Architecture
1. Data Flow
HTTP Requests → PubSub/OTLP → Span Processing → Entity Extraction → Database Triggers → Background Jobs → Issue Creation → Notifications
2. Key Components
A. Request Ingestion (src/ProcessMessage.hs)
- Receives HTTP request data via PubSub messages or OTLP protocol
- Converts to OpenTelemetry span format for unified processing
- Stores in
otel_logs_and_spanstable with computed hashes
B. Entity Extraction (src/ProcessMessage.hs:processSpanToEntities)
- Extracts from each span:
- Endpoints: Unique API routes (method + path + host)
- Shapes: Request/response structure for each status code
- Fields: Individual data fields with their paths
- Formats: Field value patterns and examples
- Uses XXHash for deterministic hash generation
- Implements caching to reduce database operations
C. Anomaly Detection (Database Layer)
- PostgreSQL triggers on entity tables:
endpoints_created_anomaly- Fires on new endpoint insertionshapes_created_anomaly- Fires on new shape insertionfields_created_anomaly- Fires on new field insertionformat_created_anomaly- Fires on new format insertionerror_created_anomaly- Fires on new error insertion
- Stored procedure
apis.new_anomaly_proc():- Creates anomaly records in
apis.anomaliestable - Batches anomalies into background jobs by project/type
- Deduplicates to prevent multiple jobs for same anomalies
- Creates anomaly records in
D. Background Job Processing (src/BackgroundJobs.hs)
- FiveMinuteSpanProcessing: Processes spans to extract entities
- OneMinuteErrorProcessing: Extracts errors from span events
- NewAnomaly: Converts anomalies to user-facing issues
E. Issue Management (src/Models/Apis/Issues.hs)
- Three issue types:
- APIChange: Groups endpoint/shape/format changes by endpoint
- RuntimeException: Individual issues for each error pattern
- QueryAlert: Threshold violations from monitoring
- Issue lifecycle: Created → Enhanced (LLM) → Acknowledged → Archived
F. Notification System (src/Pkg/Mail.hs)
- Multi-channel support: Email (Postmark), Slack, Discord, WhatsApp
- Template-based notifications with project-specific preferences
- Batched notifications to prevent spam
Implementation Details
Hash Calculation
Hashes are hierarchical to enable efficient lookups:
- Endpoint:
XXHash(projectId + host + method + urlPath) - Shape:
XXHash(endpointHash + statusCode + sortedFieldPaths) - Field:
XXHash(endpointHash + category + fieldPath) - Format:
XXHash(fieldHash + valuePattern)
Anomaly Types
-
Endpoint Anomalies
- Triggered when a new API endpoint is discovered
- Groups all related changes under single issue
-
Shape Anomalies
- Triggered when request/response structure changes
- Tracks added/removed/modified fields
-
Format Anomalies
- Triggered when field value patterns change
- Useful for detecting breaking changes in data formats
-
Runtime Exceptions
- Extracted from OpenTelemetry span events
- Groups by error type, message, and stack trace
Error Detection Process
-
Every minute, queries spans with error indicators:
- Status code = 'ERROR' or '2'
- Events containing 'exception' or 'error'
- Attributes indicating errors
-
Extracts error details from span events:
- Exception type, message, stack trace
- HTTP context (method, path)
- Service and technology info
-
Creates runtime exception issues with:
- Error grouping by hash
- Occurrence counting
- First/last seen timestamps
Issue Enhancement
- Optional LLM integration for better issue titles and recommendations
- Analyzes code changes to suggest migration complexity
- Provides actionable recommendations for developers
Database Schema
Key Tables
apis.anomalies- Raw anomaly records (legacy, being phased out)apis.issues- User-facing issues with enhanced informationapis.endpoints- Discovered API endpointsapis.shapes- Request/response structuresapis.fields- Individual data fieldsapis.formats- Field value patternsapis.errors- Runtime exceptions
Important Columns
hash- Deterministic identifier for deduplicationendpoint_hash- Links shapes/fields to endpointsissue_data- Polymorphic JSON for different issue typesllm_enhanced_at- Tracks AI enhancement status
Configuration
Project Settings
redact_fields_list- Fields to redact from stored datanotification_channels- Enabled notification methodspayment_plan- Affects rate limiting and features
Environment Variables
- Rate limits for free tier projects
- OpenAI API key for enhancement
- Notification service credentials
Testing Considerations
Current Test Coverage
ProcessMessageSpec- Basic message processingAnomalyListSpec- UI rendering (currently disabled)- Integration tests for entity extraction
Recommended Tests
- Trigger execution validation
- Error extraction edge cases
- Notification delivery confirmation
- Issue grouping logic
- Hash collision handling
Common Operations
Monitoring Anomaly Detection
-- Check recent anomalies
SELECT * FROM apis.anomalies
WHERE created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC;
-- Check background job queue
SELECT * FROM background_jobs
WHERE tag = 'NewAnomaly'
AND status = 'pending';
Troubleshooting
-
Missing Anomalies
- Check if triggers are enabled
- Verify background jobs are running
- Check for rate limiting on free tier
-
Duplicate Issues
- Verify endpoint hash calculation
- Check unique constraints on issues table
-
Missing Notifications
- Verify notification channels configured
- Check notification service logs
- Verify user email addresses
Future Improvements
-
Performance
- Implement bloom filters for hash lookups
- Optimize shape comparison algorithms
- Add caching for frequently accessed data
-
Features
- Machine learning for anomaly severity scoring
- Custom anomaly detection rules
- Webhook integrations for notifications
-
Reliability
- Add retry mechanism for failed notifications
- Implement dead letter queue for failed jobs
- Add health checks for detection pipeline