mirror of
https://github.com/monoscope-tech/monoscope.git
synced 2026-11-16 10:18:01 +00:00
- Streamline README.md from 547 to 372 lines, keeping essential Docker quick start - Create DEVELOPMENT.md with all dev setup, building, and testing content - Merge installation.md and quickstart.md into comprehensive getting-started.md - Remove duplicate content between README and dedicated docs - Move comparison table higher in README for better visibility - Improve documentation navigation with clear links between related docs This reorganization eliminates duplication while maintaining all important information in more logical locations.
5.7 KiB
5.7 KiB
Architecture Overview
System Architecture
Monoscope follows a microservices-inspired monolithic architecture, optimized for performance and ease of deployment.
┌─────────────────────────────────────────────────────────────┐
│ Ingestion Layer │
├────────────────┬────────────────┬────────────────────────────┤
│ HTTP API │ gRPC/OTLP │ Kafka Consumer │
│ (Port 8080) │ (Port 4317) │ (Optional) │
└────────┬───────┴────────┬───────┴──────────┬─────────────────┘
│ │ │
└────────────────┼──────────────────┘
│
┌─────▼─────┐
│ Router │
└─────┬─────┘
│
┌────────────────┼────────────────┐
│ │ │
┌────▼────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ Logs │ │ Metrics │ │ Traces │
│ Pipeline│ │ Pipeline │ │ Pipeline │
└────┬────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└────────────────┼────────────────┘
│
┌─────────────┴─────────────┐
│ Storage Layer │
│ (Choose One) │
├───────────┬───────────────┤
│ │ │
┌─────▼─────┐ ┌──▼──────────┐
│TimescaleDB│ │ TimeFusion │
│(PostgreSQL│ │ (Direct S3 │
│ based) │ │ Storage) │
└───────────┘ └─────────────┘
Core Components
Ingestion Layer
- HTTP API: RESTful endpoints for direct data submission
- gRPC/OTLP: OpenTelemetry Protocol support for standard telemetry ingestion
- Kafka Consumer: Optional high-throughput ingestion from Kafka topics
Processing Pipeline
- Router: Intelligently routes data to appropriate pipelines
- Log Pipeline: Processes and indexes log data
- Metrics Pipeline: Aggregates and stores time-series metrics
- Trace Pipeline: Correlates distributed traces
Storage Layer (Choose One)
- TimescaleDB: PostgreSQL-based time-series storage, keeps all data in database
- TimeFusion: Direct S3 storage engine, stores all telemetry data directly in S3 buckets for cost-effective long-term storage
AI/ML Layer
- Anomaly Detection Engine: Continuously analyzes data patterns
- Natural Language Processor: Translates natural language queries
- Pattern Recognition: Identifies trends and correlations
Technology Stack
- Language: Haskell (core application)
- Database: PostgreSQL with TimescaleDB extension
- Cache: In-memory caching for frequently accessed data
- Queue: PostgreSQL-based job queue for background tasks
- Frontend: HTMX + Alpine.js for reactive UI with minimal JavaScript
Data Flow
- Ingestion: Data enters through HTTP, gRPC, or Kafka
- Validation: Schema validation and authentication
- Enrichment: Add metadata, correlate with existing data
- Storage:
- TimescaleDB Option: Write to PostgreSQL/TimescaleDB for immediate availability
- TimeFusion Option: Write directly to S3 buckets with intelligent partitioning
- Indexing: Create indexes for fast querying
- Analysis: Background AI processing for anomaly detection
Scalability Considerations
Horizontal Scaling
- Stateless application servers can be scaled horizontally
- Load balancer distributes traffic across instances
- Shared database handles consistency
Vertical Scaling
- TimescaleDB supports partitioning for large datasets
- Read replicas for query load distribution
- Connection pooling for database efficiency
Performance Optimizations
- Batch processing for bulk ingestion
- Asynchronous job processing
- Aggressive caching strategies
- Query optimization with proper indexes
Security Architecture
- Authentication: API keys, OAuth2/Auth0 integration
- Authorization: Project-based access control
- Encryption: TLS for transport, optional encryption at rest
- Audit Logging: All API access logged for compliance
High Availability
- Database: Primary-replica setup with automatic failover
- Application: Multiple instances behind load balancer
- Storage: S3 provides built-in redundancy
- Monitoring: Self-monitoring with fallback alerting
Development Philosophy
- Simple Deployment: Single binary with minimal dependencies
- Configuration as Code: Environment variables for all settings
- Observability First: Comprehensive internal metrics
- Developer Experience: Clear APIs, good error messages