monoscope/test/integration
Anthony Alaribe 924e8e4002
Query cache (#252)
* add intelligent query cache for KQL timeseries queries

- Cache summarize queries with bin/bin_auto time binning
- Store results in UNLOGGED PostgreSQL table for performance
- Delta fetch on partial cache hits (only query new time range)
- Merge cached + new data, deduplicate by timestamp
- 24h sliding window eviction via scheduled cleanup job
- Per-project cache namespace with query hash + bin interval key

* Auto-format code with fourmolu

* fix query cache issues and improve code quality

- Fix race condition: use LEAST/GREATEST for cache time range merging
- Fix partial hit logic: handle requests extending both directions
- Add LIMIT 1 to cache lookup query
- Use existing toXXHash from Utils instead of manual xxHash call
- Single-pass stats calculation with foldl'
- Use liftA2 for Maybe operations, comparing for point-free style
- Use viaNonEmpty from Relude instead of Safe package

* add hourly cache eviction to prevent unbounded growth

- Add cleanupExpiredCache function to remove stale entries
- Evict entries where cached_to > 24h ago or last_accessed_at > 1h ago
- Integrate with HourlyJob in BackgroundJobs
- Log cleanup count when entries are deleted

* Auto-format code with fourmolu

* refactor query cache for conciseness and correctness

- Use <&> and LambdaCase for cleaner lookupCache
- Extract toPosix helper for time conversions
- Use groupBy for declarative deduplication
- Remove redundant index (UNIQUE constraint creates it)
- Increase LRU threshold to 4h (from 1h)
- Remove redundant Parser qualified import in Charts.hs

* Auto-format code with fourmolu

* fix query cache stats calculation for multi-column data

- recalculateStats now processes all value columns, not just the first
- maxGroupSum correctly sums all values per row (timestamp)
- filterByTimestamp recalculates stats after filtering
- add integration tests for cache hit, partial hit, and multi-column stats

* Auto-format code with fourmolu

* fix cache metadata mismatch and backward-extending request handling

- Use slidingWindowStart instead of entry.cachedFrom when updating cache
  after trimming, ensuring metadata matches actual data range
- Remove PartialHit case for requests extending both directions, as only
  forward delta was fetched leaving backward data missing

* optimize query cache table for performance

- Use composite primary key instead of UUID PK + unique constraint
- Remove FK constraint (orphans cleaned by LRU eviction)
- Use JSON instead of JSONB (faster writes, always read whole blob)
- Add fillfactor=70 for HOT updates on hit_count/last_accessed_at
- Add aggressive autovacuum settings for high-update table
- Simplify eviction to just LRU (remove redundant cached_to check)
- Remove unnecessary indexes (table is small, hourly eviction)

* fix query cache correctness and performance issues

- Replace LEAST/GREATEST with direct replacement in updateCache to fix
  metadata mismatch after trimOldData removes old entries
- Make sliding window configurable based on bin interval (48 points or 24h min)
- Cap mode frequency map to 1000 entries for high-cardinality data
- Use vector ifilter for deduplication instead of list groupBy conversion

* Auto-format code with fourmolu

* use Relude words and one instead of qualified imports

* refactor: simplify QueryCache code

- Simplify parseBinIntervalToSeconds using unitToSeconds helper
- Extract rowTimestamp helper to dedupe join . V.headM usage
- Replace case () with multi-way if in lookupCache

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
History 2026-01-01 14:51:17 +01:00
..
Opentelemetry Github 2-way sync with a repo (#244) 2025-12-26 01:19:12 +01:00
Pages Query cache (#252) 2026-01-01 14:51:17 +01:00
Web refactor: migrate from pg-transact to effectful-postgresql to support postgres open telemetry instrumentation (#236) 2025-12-22 22:32:43 +01:00
Main.hs deploy irrespective of the tests status 2025-09-14 22:53:50 +02:00
MonitoringSpec.hs Github 2-way sync with a repo (#244) 2025-12-26 01:19:12 +01:00
ProcessMessageSpec.hs Alerts view & Error details page (#225) 2025-12-23 17:54:20 +01:00
Spec.hs fix all failing unit tests 2025-09-04 02:28:20 +02:00