10 KiB
Anomaly Detection Testing Guide
Overview
This guide documents the comprehensive test suite created for the anomaly detection system in Monoscope. The tests verify the complete flow from HTTP request ingestion through anomaly detection, issue creation, and notification sending.
Test Structure
1. Unit Tests
ErrorExtractionSpec (test/unit/Models/Telemetry/ErrorExtractionSpec.hs)
Purpose: Tests the extraction of errors from OpenTelemetry spans
Key Test Cases:
- Error Event Extraction: Verifies that error events are correctly identified and extracted from span events
- HTTP Context Extraction: Ensures HTTP request context (method, path, host) is properly extracted from span attributes
- Hash Generation: Validates that consistent hashes are generated for similar errors
- Multiple Error Handling: Tests extraction of multiple errors from multiple spans
- Edge Cases:
- Missing events in spans
- Malformed event data
- Missing exception attributes
- Extracting HTTP path from
http.targetwhenhttp.routeis missing
Why These Tests Matter:
- Error extraction is critical for runtime exception anomaly detection
- Proper hash generation ensures deduplication of similar errors
- HTTP context is essential for grouping errors by endpoint
ProcessMessageEdgeCasesSpec (test/unit/ProcessMessageEdgeCasesSpec.hs)
Purpose: Tests edge cases in processing request messages into OpenTelemetry spans
Key Test Cases:
- Malformed JSON Handling: Tests graceful handling of invalid request/response bodies
- Missing Fields: Verifies system handles missing optional fields
- Special Characters: Tests processing of special characters in paths and parameters
- Large Payloads: Validates handling of oversized request/response bodies
- Sensitive Data Redaction: Ensures passwords, tokens, and API keys are redacted
- Cache Behavior: Tests caching of processed entities
Why These Tests Matter:
- Real-world data is often messy and incomplete
- Security is critical - sensitive data must never be stored
- Performance optimization through caching needs to work correctly
2. Integration Tests
AnomalyTriggersSpec (test/integration/AnomalyTriggersSpec.hs)
Purpose: Tests PostgreSQL database triggers that create anomalies
Key Test Cases:
- Endpoint Trigger: Verifies
endpoint_created_anomalytrigger fires on new endpoint insertion - Shape Trigger: Tests
shapes_created_anomalytrigger for new API shapes - Field Trigger: Tests field anomaly creation
- Error Trigger: Verifies error insertion triggers background jobs (but not anomaly records)
- Background Job Batching: Tests that multiple anomalies are batched into single job
- Job Updates: Verifies existing jobs are updated rather than creating duplicates
Why These Tests Matter:
- Database triggers provide 100% coverage - they can't be bypassed
- Batching reduces background job queue pressure
- Job deduplication prevents processing the same anomaly multiple times
AnomalyDetectionFlowSpec (test/integration/AnomalyDetectionFlowSpec.hs)
Purpose: End-to-end integration tests for the complete anomaly detection flow
Key Test Cases:
-
New Endpoint Detection:
- Send HTTP request → Process into span → Extract entities → Trigger creates anomaly → Background job creates APIChange issue
- Verifies the complete flow from request to issue creation
-
Multiple Anomaly Grouping:
- Single request with complex JSON creates multiple entities (endpoint, shape, fields)
- All anomalies grouped into single APIChange issue
- Tests issue data contains all anomaly information
-
Runtime Exception Detection:
- OpenTelemetry span with error event → Process errors → Create error record → Background job creates RuntimeException issue
- Validates error extraction and issue creation
-
Issue Updates:
- First request creates issue → Second request with changes updates same issue
- Tests that issues are reused, not duplicated
Why These Tests Matter:
- Validates the entire system works end-to-end
- Ensures anomalies are properly grouped to avoid notification spam
- Confirms issue updates work to track ongoing changes
AnomalyNotificationsSpec (test/integration/AnomalyNotificationsSpec.hs)
Purpose: Tests the multi-channel notification system
Key Test Cases:
- Endpoint Anomaly Notifications: Tests notifications for new API endpoints
- Runtime Error Notifications: Validates error notifications with full context
- Notification Batching: Multiple anomalies for same endpoint send single notification
- Channel Preferences: Respects project settings for enabled channels (Email, Slack, Discord, WhatsApp)
- Notification Content: Verifies correct data in notification payloads
Why These Tests Matter:
- Notifications are the user-facing output of the system
- Batching prevents notification spam
- Multi-channel support allows flexible alerting
AnomalyListSpec (test/integration/Pages/Anomalies/AnomalyListSpec.hs)
Purpose: Tests the anomaly list web UI endpoint
Key Test Cases:
- Issue Listing: Retrieves and displays issues correctly
- Acknowledgment Flow: Tests acknowledging anomalies marks them as reviewed
- Hierarchical Display: Parent anomalies hide children until acknowledged
- Issue Type Filtering: Proper filtering by APIChange vs RuntimeException
Why These Tests Matter:
- UI is how users interact with detected anomalies
- Acknowledgment workflow prevents re-alerting
- Hierarchical display reduces UI clutter
3. Test Utilities
AnomalyTestUtils (test/Pkg/AnomalyTestUtils.hs)
Purpose: Shared test utilities and helpers
Key Components:
- Entity Creators: Functions to create test endpoints, shapes, fields, errors
- Verification Helpers: Check if anomalies, issues, and jobs were created
- Data Cleanup: Clear test data between runs
- Span Builder DSL: Fluent interface for creating test OpenTelemetry spans
- Background Job Helpers: Process jobs and wait for completion
Testing Flow Rationale
Why These Specific Flows?
-
HTTP Request → Anomaly Detection:
- This is the primary use case - detecting API changes from live traffic
- Tests verify each step: ingestion → entity extraction → anomaly creation → issue generation
-
Error Detection from Traces:
- Modern observability relies on OpenTelemetry
- Errors in traces should automatically create issues
- Tests ensure error context is preserved
-
Anomaly Grouping:
- Individual field changes would create too many notifications
- Grouping by endpoint provides the right level of abstraction
- Tests verify grouping logic works correctly
-
Multi-Channel Notifications:
- Different teams prefer different notification channels
- Tests ensure all channels work and respect preferences
Database Schema Assumptions
The tests assume this schema:
apis.endpoints: Stores API endpointsapis.shapes: Request/response structures per endpointapis.fields: Individual fields within shapesapis.formats: Field value patternsapis.errors: Runtime errorsapis.anomalies: Individual detected changesapis.issues: User-facing grouped anomaliesbackground_jobs: Async job queue
TODO List for Recreating Tests
Phase 1: Setup
- Create test database with proper schema
- Set up test fixtures and data generators
- Configure test environment variables
- Create
AnomalyTestUtilsmodule with helper functions
Phase 2: Unit Tests
-
Create
ErrorExtractionSpec:- Test error event filtering from spans
- Test HTTP context extraction
- Test hash generation consistency
- Test malformed data handling
- Test technology stack extraction
-
Create
ProcessMessageEdgeCasesSpec:- Test malformed JSON handling
- Test missing field handling
- Test special character handling
- Test large payload truncation
- Test sensitive data redaction
- Test entity caching
Phase 3: Integration Tests
-
Create
AnomalyTriggersSpec:- Test endpoint creation trigger
- Test shape creation trigger
- Test field creation trigger
- Test error insertion trigger
- Test background job batching
- Test job update logic
-
Create
AnomalyDetectionFlowSpec:- Test new endpoint detection flow
- Test multiple anomaly grouping
- Test runtime exception detection
- Test issue update mechanism
- Test complete end-to-end flow
-
Create
AnomalyNotificationsSpec:- Test Email notifications
- Test Slack notifications
- Test Discord notifications
- Test WhatsApp notifications
- Test notification batching
- Test channel preferences
-
Create
AnomalyListSpec:- Test issue retrieval
- Test acknowledgment flow
- Test hierarchical display
- Test filtering and pagination
Phase 4: Test Infrastructure
- Configure test database cleanup between runs
- Set up parallel test execution
- Add test coverage reporting
- Create CI pipeline configuration
- Document how to run tests locally
Phase 5: Additional Tests to Consider
- Performance tests for high-volume anomaly processing
- Stress tests for background job system
- Tests for LLM enhancement of issue titles
- Tests for anomaly approval workflow
- Tests for query alert anomalies
- Tests for webhook delivery
Key Testing Principles Applied
- Test at Multiple Levels: Unit tests for logic, integration tests for flows
- Test Real Scenarios: Based on actual usage patterns
- Test Edge Cases: Malformed data, missing fields, large payloads
- Test Security: Ensure sensitive data is redacted
- Test Performance: Caching and batching mechanisms
- Test User Experience: Notifications and UI interactions
Running the Tests
Prerequisites
- PostgreSQL database with test schema
- Proper environment variables set
- All Haskell dependencies installed
Commands
# Run unit tests
make test-unit
# Run integration tests
make test-integration
# Run all tests
make test
Maintenance Notes
- Database Triggers: If triggers change, update
AnomalyTriggersSpec - New Anomaly Types: Add test cases for new types in flow tests
- New Notification Channels: Add to
AnomalyNotificationsSpec - Schema Changes: Update test data generators in
AnomalyTestUtils