Commit graph

200 commits

Author SHA1 Message Date
Anthony Alaribe
64b3a8bc88
refactor: migrate from pg-transact to effectful-postgresql to support postgres open telemetry instrumentation (#236)
* refactor: migrate from pg-transact to effectful-postgresql

Replace pg-transact and pg-transact-effectful with effectful-postgresql for database operations. Key changes: replace DB effect with WithConnection, remove dbtToEff wrappers, use direct PG.query/PG.execute calls, update runDB to runWithConnectionPool, keep pg-entity for query building, add IOE constraint alongside WithConnection where needed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* Auto-format code with fourmolu

* chore: use effectful-postgresql fork with OpenTelemetry support

Switch to tonyalaribe/effectful-postgresql fork (instrument-pg branch) and
enable the +enable-otel flag for OpenTelemetry instrumentation.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* fix: improve connection usage and error handling in BackgroundJobs

- Use PG.execute directly instead of multiple withConnection calls for
  CleanupDemoProject (more efficient connection usage)
- Handle missing project gracefully in ATRuntimeException processing
  instead of crashing with error
- Clean up unused imports (PGS, withConnection)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* fix: correct effectful-postgresql subdir path

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* fix: add proper error handling and logging for critical operations

- Log advisory lock release failures instead of silently ignoring
- Log when error update doesn't match any span (potential data issue)
- Log when span hash update count mismatches expected (data integrity)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* Auto-format code with fourmolu

* refactor: simplify single-row query handling in BackgroundJobs

* refactor: return lists from model query functions instead of Vectors

PostgreSQL-simple returns lists naturally, so wrapping results with V.fromList was wasteful. Model query functions now return lists directly, with Vector conversion happening at call sites that actually need Vectors.

* Auto-format code with fourmolu

* perf: fix N+1 query pattern and improve advisory lock handling

- Replace individual UPDATE queries in forM_ loop with batched UPDATE
  using SQL unnest to reduce O(n) DB round-trips to O(1)
- Add 5-minute statement timeout to advisory lock to prevent hangs
- Add timeout reset in finally block for cleanup
- Improve comments explaining session-level lock auto-release behavior

* refactor: replace verbose Only unwrapping with coerce

- Use coerce @[Only T] @[T] instead of map (\(Only x) -> x)
- Use coerce @(Maybe (Only T)) @(Maybe T) instead of fmap (\(Only x) -> x)
- Update GenerateOtelFacetsBatch to use Projects.ProjectId instead of Text

* Auto-format code with fourmolu

* fix: advisory lock release and remove unnecessary vector conversion

- Use query instead of execute for pg_advisory_unlock (SELECT returns result)
- Add validation for unexpected unlock results
- Remove unnecessary V.fromList/V.toList round-trip in runHourlyJob

* refactor: simplify COUNT(*) query result handling

Use inline maybe/listToMaybe instead of case expression for cleaner code.

* Auto-format code with fourmolu

* refactor: use catch instead of try/case for advisory lock cleanup

Cleaner error handling with catch pattern instead of explicit try + case Left/Right.

* Auto-format code with fourmolu

* perf: use V.mapMaybe and V.unzip3 to avoid list conversions

Direct Vector operations for batch error updates instead of
list mapMaybe + V.fromList.

* refactor: introduce DB type alias and switch to pg-transact

* Auto-format code with fourmolu

* fix errors in specs

* fix: remove duplicate import in Dashboards.hs

* fix: resolve integration test failures

- BusinessFlowsSpec: fix column names for apis.subscriptions table

- TestUtils: add test user as admin to nil UUID project

- System/Types: increase static UUID pool to prevent collisions

- AnomaliesSpec: fix test assertions for correct filter views

* fix aroundAll

* fix: resolve integration test failures

- Fix insertProjectMembers to handle empty list (prevents 'no rows supplied' error)

- Fix DashboardsSpec UUID collisions by grouping creations and adding cleanup

- Fix ManageMembersSpec: correct userID to UUID 1, update expected member count and email

- Fix ProjectsSpec: update expected project count to include demo project

* remove PG_EFFECT_MIGRATION.md

* fix: use unique team handles in bulk add teams test to avoid conflicts

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-22 22:32:43 +01:00
Anthony Alaribe
5cd948697f
Teams improvement v2 (#234)
* Improve dashboards and teams UI with star functionality

- Add star/unstar functionality for dashboards with persistent state
- Update dashboard sorting to prioritize starred items
- Improve table component styling and layout
- Add icons for dashboards and teams in list views
- Streamline table columns and remove unnecessary checkboxes
- Update bulk actions to use new table component features

* Auto-format code with fourmolu

* Fix teams bulk actions to use itemId field

Update TBulkActionForm to use itemId instead of teamId to match
the Table component's standardized field name for bulk actions.

* Improve settings navigation layout and styling

- Reduce navigation width for better space utilization
- Update font sizes and spacing for cleaner appearance
- Improve icon styling with background colors
- Enhance navigation item padding and layout

* Refactor API keys page to use Table component and add client-side tab support

- Enhanced TabFilter component to support both client-side and server-side tabs
- Added clientSide boolean field to TabFilter for tab switching mode
- Added targetId field to TabFilterOpt for custom element targeting
- Refactored API keys page to use reusable Table component
- Updated all existing TabFilter usage to include new fields
- Improved consistency across table implementations

* Auto-format code with fourmolu

* nitpicks on

* add test for starring and unstarring, and dashboard default sorting

* Fix tagify scope issue in dashboard creation modal

The tagify variable was scoped inside DOMContentLoaded event listener,
causing a reference error when getSelectedTeams() tried to access it
during form submission. Moved declaration to outer scope.

* Fix gridStackInstance null error when adding dashboard widgets

* Fix teams badge WCAG contrast by using badge-neutral

* Fix compilation errors

- Fix do-notation errors by adding proper do blocks instead of using >> after $
- Rename local section_ helper to teamSection_ to avoid shadowing
- Remove unused ProjectId parameter from renderMonitorContent

* Use brackets instead of do-blocks for single-line expressions

- Keep code on one line by using parentheses and >> operator
- Applied to card_, notifRow_, lazySection_, teamSection_, field_, fieldIcon_ helpers

* Apply code formatting updates

- Replace font-mono with monospace class
- Minor whitespace adjustments

* Fix duplicate html assignment bug and simplify code

* Auto-format code with fourmolu

* Refactor tables to use standardized Table component with sorting

- Add containerId to Table Config for HTMX targeting
- Add renderAsTable mode for proper table rendering
- Add rowId and rowAttrs to TableRows for load-more consistency
- Fix th elements to include column attrs for alignment
- Fix class concatenation spacing in table headers
- Fix scrolling in BodyWrapper (overflow-y-hidden to overflow-y-auto)
- Add sort parameter to Issues.selectIssues
- Refactor Anomalies and ApiCatalog to use TableHeaderActions
- Remove flex classes from table cells

* Auto-format code with fourmolu

* Fix table load more by moving pagination inside tbody

- Move pagination link inside tbody for table mode (as a <tr>)
- Add renderTablePaginationLink with hx-target='closest tr'
- Add id to tbody for targeting
- Pagination row gets replaced with new rows + new pagination row

* Refactor Anomalies table to use built-in checkbox

- Remove custom checkbox column (use table's rowId feature instead)
- Add status indicator dot to Issue column title row
- Update TableRows to use proper rowId for load more

* Auto-format code with fourmolu

* Fix Anomalies issue column layout structure

- Fix indentation so metadata, stats, actions are separate blocks
- Status indicator dot in title row
- Condense code using single-line patterns where appropriate

* Auto-format code with fourmolu

* Merge renderPaginationLink and renderTablePaginationLink into single function

* Auto-format code with fourmolu

* Align row checkboxes to top and add vertical padding to issue column

* Improve settings sidebar: larger text, neutral icons, stronger selection highlight

* Widen settings sidebar

* Fix hlint warnings

* Fix all hlint warnings across codebase

* Auto-format code with fourmolu

* Fix GHC warnings: dodgy import, name shadowing, unused do-bind

* Use foldMap and zipWithM_ for cleaner iteration patterns

* Extract columnCount helper and add error handling for processVariable

* Auto-format code with fourmolu

* Optimize URL construction and use single-pass fold for change counting

* Extract issueTypeBadge helper to reduce code duplication

* Fix SQL injection vulnerabilities and semantic error in change counting

* Auto-format code with fourmolu

* Simplify SQL validation and remove redundant variable

* Auto-format code with fourmolu

* Rename selectDashboardsSorted to selectDashboardsSortedBy for clarity

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-12-19 23:44:55 +01:00
Anthony Alaribe
6b0c5c8111 remove unused code 2025-11-27 03:09:22 +01:00
Anthony Alaribe
4b0e92659b Fix integration test compilation errors after testServant refactoring
Update all integration tests to properly unpack tuple response from testServant helper, which now returns (RespHeaders a, a).

Key changes:
- Add servant and servant-htmx to integration test dependencies
- Fix tuple unpacking in all test specs (AnomalyList, Log, Alerts, ManageMembers, Projects, BusinessFlows, Reports, Api, ApiCatalog, GrpcIngestion)
- Fix BusinessFlowsSpec imports to use ResponseHeader data constructor
- Fix type-level string usage for header lookups
- Fix ProjectId to Text conversions using .toText
- Add missing pattern match cases to avoid incomplete pattern warnings
2025-11-17 11:30:07 +01:00
Anthony Alaribe
4cddf7f82e Update OpenAI integration to use new API format
- Add openai package dependency
- Update LLM effect to use OpenAI V1 chat completion parameters
- Pass model name and messages via proper parameter structure
- Fix error handling to use show instead of toText
2025-11-15 20:04:18 +00:00
Anthony Alaribe
28a250766c Add gRPC ingestion test suite and update test database setup
- Add GrpcIngestionSpec to test suite in package.yaml and cabal
- Add required dependencies: lens, proto-lens, grapesy
- Update Makefile timescaledb-docker-tmp to use faketime for frozen test time
2025-11-15 05:59:12 +00:00
Anthony Alaribe
8a08da4b46 optimize span processing performance and add tracing instrumentation
- Replace O(n²) nubBy with O(n) HashMap-based deduplication for endpoints, shapes, fields, and formats
- Batch UUID generation upfront to avoid repeated IO operations per span
- Make entity extraction pure computation by pre-generating all UUIDs
- Add OpenTelemetry tracing spans for pubsub and kafka batch processing with error tracking
- Add hashable dependency for HashMap-based deduplication
- Remove unnecessary threadDelay in test utilities
- Add strictness annotations to improve memory usage

These changes significantly reduce the algorithmic complexity and IO overhead of span processing while improving observability.
2025-11-15 05:59:07 +00:00
Anthony Alaribe
5d9ca23155 expand time widget logic fixes and suggestions dropdown test fixes 2025-10-26 16:00:13 +01:00
Anthony Alaribe
f6707211f0 add missing dev dependencies so tests build 2025-09-29 21:46:11 +02:00
Anthony Alaribe
2f32accb97
Performance optimization (#196)
* use details and summary for presets dropdown.

* tiny query lib improvements

* improve the list better

* allow deletes remove and update list via htmx

* x

* reduce dom count of the facets elements.

* checkpoint

* fix not collapsing parent without children

* remove codemirrow

* enable source maps, update echarts, update tailwind, etc

* cleanup listeners and be more lazy in session reploay

* onboarding
2025-09-20 02:07:33 +02:00
Yussif Mohammed
9741ec685a Revert "update name to monoscope-server"
This reverts commit 4fcda41935.
2025-09-15 21:27:43 +00:00
Yussif Mohammed
4fcda41935 update name to monoscope-server 2025-09-15 16:05:29 +00:00
Anthony Alaribe
32337fd7ad disable non haskell files from the source files 2025-09-14 20:55:13 +02:00
Anthony Alaribe
4b4ed0252a rename project to monoscope 2025-09-14 19:55:26 +02:00
Anthony Alaribe
b6711269aa
Support Basic auth (For opensource users) and investigate performance issue in onboading due to returning 401 unauthenticated, causing repeated authentication checks. (#193)
* support for basic auth for open source users

* implement basic auth support

* introduce support for basic auth and support disabling the client side monitoring via monoscope-browser

* pass in the entire config to bodywrapper

* support leaving unused fields as optional

* make fmt

* add posthog

* configure external scripts via env variables

* make fmt

* fix compile errors
2025-09-12 17:06:32 +02:00
Anthony Alaribe
0ac389984a fix test compilation error due to the byte libraris. would have to remove one of the libraries soon 2025-09-05 03:18:52 +02:00
Anthony Alaribe
da32b25a07 fix and make doctests compile and pass 2025-09-04 22:22:46 +02:00
Anthony Alaribe
44e294274a fix all failing unit tests 2025-09-04 02:28:20 +02:00
Anthony Alaribe
a4706a09f5 fix issue preventing compilation of some yamls in dashboards 2025-09-04 02:28:20 +02:00
Yussif Mohammed
ec6c20a228
Replay sessions (#183)
* replay events and publish/receive from/to pubsub

* pubsub for rrweb events works

* allowing CORS for all origins in OpenTelemetry

* fix: allow messages with project IDs

* session ui

* session replay modal

* towards fixing hls

* revert

* feat: save session files

* remove unused save file variants

* fetch and serve replay events in ui

* use background ctx

* remove amazonka dependency

* update yamr loc

* format
2025-07-31 09:46:49 +00:00
Anthony Alaribe
f61b0881d8 pass context spans around the application better. 2025-07-15 17:06:15 +02:00
Anthony Alaribe
a87bf1f42c improve tracing and integrate more observability 2025-07-15 16:01:24 +02:00
Anthony Alaribe
bfb57e79fd fix-ci: statically link libs. 2025-07-15 12:56:25 +02:00
Anthony Alaribe
0dadb00fff fix-ci: statically link libs. so it works in docker container 2025-07-15 11:54:08 +02:00
Anthony Alaribe
782d143214 fix-ci: include static in package assets 2025-07-14 23:40:34 +02:00
Anthony Alaribe
643e1a8c20 fix-ci: include static in package assets 2025-07-14 23:29:22 +02:00
Anthony Alaribe
47da7cc6f3 fix-ci: include autoinstrumentation config toml 2025-07-14 22:46:08 +02:00
Anthony Alaribe
14acf9f969 add build tool depends 2025-07-13 23:05:57 +02:00
Anthony Alaribe
3f15832688 suppress compile errors from mockdata 2025-07-10 01:35:20 +02:00
Anthony Alaribe
8b033fac28 improve api catalog logic and make tests pass well. also remove the time duration between when anomalies are found and when the issues are processed 2025-07-07 00:21:04 +02:00
Anthony Alaribe
f5ddc11f5f unit tests pass. checkpoint towards passing integration tests 2025-07-03 15:15:11 +02:00
Anthony Alaribe
524d71c095 test in the github actions as separate piepline flows 2025-07-02 01:41:16 +02:00
Anthony Alaribe
f228b96303 fix unit tests setup. now passing 2025-07-02 01:41:15 +02:00
Anthony Alaribe
b6d2bc8314 remove unused dependencies for unit tests 2025-07-02 01:41:15 +02:00
Anthony Alaribe
4dd34754ff move unit tests and integration tests to their own project s they run quickly 2025-07-02 01:41:15 +02:00
Anthony Alaribe
1b5ef9a865 enable processing events in gcp and refactor and consolidate logItem logic to one function and multiple 2025-06-26 14:51:13 +02:00
Anthony Alaribe
799c719b01 checkpoint 2025-06-26 12:59:50 +02:00
Anthony Alaribe
e3056b7805 improve processing of events. cache project keys lookup, and allocate little memory. now processing 200 events in 1 second. 2025-06-26 11:23:38 +02:00
Anthony Alaribe
515fd2fb2c fix regex precision. todo: improve and fix test suite 2025-06-23 03:51:46 +02:00
Anthony Alaribe
330e40b923 improve regex and formats logic to be more general and ideally not create an error or anomaly for every single annomaly 2025-06-22 02:31:50 +02:00
Anthony Alaribe
eb7c2c47e9 fix lint errors and unused imports warnings 2025-06-19 05:27:45 +02:00
dawkaka
93808ac9d4 discord bot 2025-06-09 14:45:22 +00:00
Anthony Alaribe
b9a2e120d8
summarize by, query langauge support to match the kql language (#176)
* fix completion issue on query editor

* swap agg and by

* checkpoint agg works

* checkpoint group by

* group by with functinoing binning ui

* checkpoint: figure out sort by input

* checkpoint

* checkpoint

* checkpoint

* checkpoint

* show and hide sort by limit and more correctly

* generating new sumamrize by syntax

* support sumamry by

* checkpoint

* checkpoint

* modify dash temps

* anomaly list

* page chart widgets

* stick to kql as much as possible

* sumarize

* sumarize

* sumarize

* sorta functional

* functioning aggs
2025-06-04 05:33:23 +02:00
Anthony Alaribe
e482436075
New editor and AI query support (#172)
* checpoint working editor. need to submit query and actually search

* working handleAddQuery impl

* ask ai btn

* llm search checkpoint before integrate langchain

* working ai search checkpoint

* chekcpoint not overlapping suggestions and can search

* checkpoint

* flatter schemas support

* prettier config

* use ? for ai queries

* checkpoint. TODO: include schema to ai search with a schema resolution endpoint

* move schema to backend

* widget fixes to explorer

* register popular queries
2025-05-25 07:53:20 +02:00
Mahmud Bello
0aad242242
fix : ffi linker issue for linux resolved in ghc 9.10.22 (#166)
* fix : ffi linker issue for linux resolved in ghc 9.10.22

* fix : stack resolver updated for workflow

* Revert "fix : ffi linker issue for linux resolved in ghc 9.10.22"

This reverts commit c2ef849d93.

* fix : stack resolver updated for workflow

* fix : corrected mistakes in versioning

* add libkafka to workflow

* fix: correct workflow error

* fix: correct workflow error

* add hs otel to tests

---------

Co-authored-by: = <=>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
2025-05-15 04:32:14 +02:00
Anthony Alaribe
b526047d9b fix deep merging of objects 2025-04-22 22:35:20 +02:00
Anthony Alaribe
b7e3249f43 gogol updated and now broken. brought back facets filter 2025-04-21 23:34:28 +02:00
Anthony Alaribe
31337c7fcf functioning otel via http 2025-04-20 20:54:19 +02:00
Anthony Alaribe
23856b303e
Merge branch 'master' into otel 2025-04-19 01:50:52 +02:00
Anthony Alaribe
7181f7a089 remove microlens package as its not needed 2025-04-18 20:19:17 +02:00