Commit graph

16 commits

Author SHA1 Message Date
Anthony Alaribe
014e0ca2af
Kql update (#276)
* Add KQL scalar functions, SQL preview, and dashboard YAML editing

- Add scalar function support: coalesce, iff, isnull, isnotnull, toint, tofloat, tostring, round
- Fix extend column resolution for GROUP BY clauses
- Separate aggregations from extended columns in query components
- Add SQL debug preview endpoint for widgets
- Add dashboard YAML get/put endpoints for editing
- Show empty constants warning on dashboard pages
- Fix ORDER BY to use GROUP BY column when timestamp not in GROUP BY
- Add yaml-editor web component

* Auto-format code with fourmolu

* Refactor Widget.hs and DashboardUtils.hs for cleaner code

- Hoist V.toList dataRows conversion in renderTableWithDataAndParams
- Simplify KQL remapping using mapMaybe in variablePresetsKQL
- Replace forM_ (zip xs [0..]) with ifor_ for indexed iteration
- Add TODO comments for rawQuery/rawSql Phase 4 removal
- Add doctests for replacePlaceholders, constantToSQLList, constantToKQLList

* Auto-format code with fourmolu

* switch list to table in dashboard

* Add KQL arithmetic expressions and fix parser issues

- Add arithmetic support in summarize: count() / 5.0, count() * 100 / sum(x)
- Fix by-clause parsing to distinguish subjects from scalar functions
- Add ByClauseItem type for proper by-clause item representation
- Fix non-bin summarize queries to use aggregations in SELECT
- Fix extend operator to append columns to defaults (not replace)
- Add extendSelect field to QueryComponents for proper extend semantics
- Forward dashboard constants via data-constants attribute in HTMX
- Use variablePresetsKQL for KQL query expansion in chart endpoint

* Fix KQL arithmetic parsing and refactor SQL generation

- Fix arithmetic operator precedence using makeExprParser
- Fix numeric literal parsing order (try pNumericOnly before aggFunctionParser)
- Add try to arithmetic operators for proper backtracking with 'by' clause
- Add NormalizedQuery type for cleaner SQL generation
- Add transformAST for AST normalization (combineWheres, reorderSections)
- Add unit test for arithmetic division with bin_auto()
- Add doctest for percentiles with by clause regression

* Auto-format code with fourmolu

* Fix hlint warnings in Parser modules

* Performance and safety improvements for KQL parser and dashboards

- Fix KQL remapping bug: filter out -kql suffix entries from output
- Add time interval bounds checking (max 1 year) to prevent DoS
- Use Map.lookup instead of list lookup in resolveExtendedColumn (O(log n))
- Use Set for flattenedOtelAttributes for O(log n) membership checks
- Pre-compute replacement maps once in mkWidgetProcessor for efficiency

* Auto-format code with fourmolu

* Fix regex recompilation and restore rawQuery for Monaco editor

- Pre-compile placeholder regex at module load for performance
- Restore rawQuery field in Widget to preserve original query with placeholders
- Use rawQuery in widget editor for Monaco to display original KQL query

* Improve dashboard widget search and test assertions

- Fix findWidgetInDashboard to recursively search into children
- Process dashboard constants concurrently with pooledForConcurrently
- Simplify variablePresetsKQL using list comprehension
- Update ParserSpec tests to compare full SQL and error messages
- Remove T.isInfixOf partial matches in favor of exact comparisons

* Fix regex type safety and add argument validation to KQL functions

- Use Text instead of String in placeholder regex for type consistency
- Add argument count validation for iff, isnull, isnotnull, isempty, isnotempty
- Fix round() decimal validation logic (was inverted)
- Update doctest expected output for percentiles with by clause

* Optimize Widget rendering and add placeholder DoS protection

- Use Vector operations directly instead of converting to lists
- Add 1MB size limit to placeholder replacement to prevent DoS
- Refactor scalar function SQL generation to use Maps
- Fix empty constants check to include Nothing case

* simplify

* Fix widget query loss and SQL preview not loading

- Fix SQL preview: use 'toggle from:closest details' trigger since toggle
  event fires on parent details element, not child divs
- Fix SQL preview: use raw_query || query (snake_case JSON field names)
- Fix query loss: add mergeWidgetPreservingQuery to preserve original query
  and rawQuery fields when updating widgets instead of replacing entirely

* Fix stat widget color and add variable defaults to constants

- Fix timeseries_stat widgets showing green instead of blue by using
  default palette color for generic column names (value, count, etc.)
- Add addVariableDefaults to ensure dashboard constants can reference
  variables like {{var-resource}} even when not in URL params

* Fix widget duplication and deletion in dashboards

- Fix widget duplication to properly initialize charts and scripts
- Use template element to extract widget HTML and re-query DOM after GridStack adds it
- Process htmx, hyperscript, and scripts on the live DOM element
- Fix widget deletion to use closest grid-stack instead of global instance
- Fix hyperscript syntax for drawer close event

* Auto-format code with fourmolu

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-09 23:38:28 +01:00
Anthony Alaribe
9d4831f05e
Add KQL operators: extend, project, round, tofloat, toint, tostring (#275)
* Add KQL operators: extend, project, round, tofloat, toint, tostring

New KQL features:
- extend operator: adds computed columns to results
- project operator: selects and renames specific columns
- round(value, decimals): round to N decimal places
- tofloat/todouble: convert to float
- toint/tolong: convert to integer
- tostring: convert to text

Updated documentation with examples and usage notes.

* Fix ToQueryText for AggFunction to produce KQL instead of SQL

- ToQueryText AggFunction now produces valid KQL syntax that can be
  parsed back, fixing query round-trip for extend/project commands
- Add ToQueryText instance for SubjectExpr
- Consolidate type casting parsers (pToFloat, pToInt, pToString) with
  pTypeCast helper, reducing duplication
- Consolidate extend/project parsers with pColumnCommand helper

* Reduce parser duplication and add round() validation

- Unify pColumnAssignment and namedAggregation into pNamedExpr helper
- namedAggregation now delegates to pNamedExpr (snd <$> pNamedExpr)
- Add validation that round() decimals parameter must be numeric
- Remove outdated doctest for round with arithmetic expression

* Improve parser robustness and add error validation tests

- Add eof to parseQuery to require consuming all input
- Remove try from pCoalesce and pRound to propagate validation errors
- Add sanitizeAlias to replace dots with underscores in SQL aliases
- Add error tests for dcount accuracy, coalesce args, round decimals
- Update test expectations to match new alias output format

* Support nested aggregations in type-casting functions

Add ScalarExpr type to enable patterns like round(countif(...), 2) or
tofloat(sum(field)). Type-casting functions (round, tofloat, toint,
tostring) now accept either values or nested aggregations.

- Add ScalarExpr (SVal | SAgg) with Display/ToQueryText instances
- Add aggToSqlNoAlias for SQL generation without AS alias
- Extract baseAggParsers for reuse in nested parsing
- Add pScalarExprNested parser for nested expressions
- Add getAlias helper to extract explicit aliases

* Improve Stats parser: generic-lens, helpers, validation, tests

- Extract simpleAggSQL and percentileSQL helpers to reduce duplication
- Replace getAlias/setAlias with generic-lens typed accessors
- Allow dots in pNamedExpr names (sanitized to underscores)
- Add round() decimals validation (0-15 per PostgreSQL limit)
- Add tests for nested aggregations, project SQL, round validation

* Add integration tests for nested aggregation SQL generation

- Full SQL verification for round(countif(...), N)
- Tests for tofloat(sum()), toint(count()) nested patterns
- Tests for extend/project with nested aggregations
- Tests for named nested aggregations preserving aliases
- Minor cleanup: use {} pattern syntax in defaultAlias

* Auto-format code with fourmolu

* Support negative numbers in parser for round() validation

Use L.signed wrapper for L.float and L.decimal to parse negative
numbers like -1, enabling round() decimals validation to properly
reject negative values.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-08 15:26:32 +01:00
Anthony Alaribe
69b56958c4
Create KQL operators documentation (#267)
* Add comprehensive KQL reference documentation

Document all KQL operators and functions supported in Monoscope:
- Comparison, string, membership, and logical operators
- Pipe operators (where, summarize, sort, take)
- Aggregation functions including percentiles and conditional counts
- Scalar functions (strcat, coalesce, iff, case)
- Date/time functions (now, ago, bin, bin_auto)
- Field reference syntax including nested JSON and OTel attributes
- Extensive query examples for common use cases

* Add KQL-standard default column aliases for aggregations

- Generate default aliases like count_, sum_field when no explicit alias provided
- Update Display instance to include AS alias clause in generated SQL
- Document default column naming conventions in KQL reference
- Clarify matches/=~ as syntax variants for regex matching

* Improve KQL documentation clarity and completeness

- Clarify has vs contains operators with examples
- Fix bin_auto() time range table notation
- Add Limitations & Notes section covering:
  - Unsupported KQL features
  - Null handling behavior
  - Type coercion rules
  - Performance tips

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-07 20:02:47 +01:00
Anthony Alaribe
9527fa7571 Reorganize project structure
Move config files to config/ directory (tailwind.config.js, workbox-config.js, weeder.toml). Move documentation to docs/ (DEVELOPMENT.md, DOCKER_COMPOSE_README.md). Update all references in Dockerfile, Makefile, tailwind.css, and documentation.
2025-12-17 00:49:42 +01:00
Anthony Alaribe
4791a652b7 Swap gRPC and HTTP order in example 2025-11-26 17:54:52 +01:00
Anthony Alaribe
1ffd869dfe Improve logs+traces email copy with concrete example 2025-11-26 01:15:14 +01:00
Anthony Alaribe
eebed83cf6 Update email sequence URLs to monoscope.tech domain 2025-11-26 00:57:25 +01:00
Anthony Alaribe
39503d7be1 Add Monoscope logo assets for email sequence 2025-11-26 00:36:21 +01:00
Anthony Alaribe
ab4c029bcc Refine email sequence copy and update logos
- Replace em dashes with periods for more natural tone
- Add (n/6) numbering to all drip email subjects
- Broaden messaging to cover full observability (logs, traces, metrics, APIs)
- Update logo placeholders to monoscope.png
- Use colons instead of em dashes in bullet points
2025-11-26 00:28:00 +01:00
Anthony Alaribe
244152ae09 update email copy and add dependabot config 2025-11-26 00:01:03 +01:00
Anthony Alaribe
8bd1c2621b email sequence initial trial 2025-11-25 23:45:12 +01:00
Anthony Alaribe
e95e6b69da Refactor documentation structure for better organization and clarity
- Streamline README.md from 547 to 372 lines, keeping essential Docker quick start
- Create DEVELOPMENT.md with all dev setup, building, and testing content
- Merge installation.md and quickstart.md into comprehensive getting-started.md
- Remove duplicate content between README and dedicated docs
- Move comparison table higher in README for better visibility
- Improve documentation navigation with clear links between related docs

This reorganization eliminates duplication while maintaining all important information
in more logical locations.
2025-11-23 22:41:44 +01:00
Anthony Alaribe
4b4ed0252a rename project to monoscope 2025-09-14 19:55:26 +02:00
Anthony Alaribe
3c760312be testing guide 2025-09-08 20:47:05 +02:00
Anthony Alaribe
25c6dc639b document more of the anomaly detection process 2025-09-08 20:47:05 +02:00
Anthony Alaribe
df1dac614d enable -Weverything ghc options and fixing code to remove some warnings 2022-02-24 01:23:08 +01:00