mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
Adds a complete v-next observability storage adapter for ClickHouse,
providing full feature parity with the existing DuckDB adapter across
all five signal
domains.
- Tracing — createSpans, getSpan, getTraceSpans with
ReplacingMergeTree-based dedup and LIMIT 1 BY for latest-version span
retrieval
- Trace roots — getTraceRoots, getTraceById with paginated trace listing
and full hierarchy fetch
- Logs — createLogs, listLogs with filter/pagination support
- Metrics — createMetrics, listMetrics plus OLAP analytics
(getMetricAggregate, getMetricBreakdown, getMetricTimeSeries,
getMetricPercentiles) with
label-based groupBy and cost tracking (getCostSummary)
- Scores — createScore, batchCreateScores, listScores plus OLAP
analytics (getScoreAggregate, getScoreBreakdown, getScoreTimeSeries,
getScorePercentiles)
keyed by scorerId
- Feedback — createFeedback, batchCreateFeedback, listFeedback plus OLAP
analytics (getFeedbackAggregate, getFeedbackBreakdown,
getFeedbackTimeSeries,
getFeedbackPercentiles) keyed by feedbackType with numeric valueNumber
aggregation
- Discovery — getEntityTypes, getEnvironments, etc. via pre-computed
refreshable materialized views
- DDL — 8 tables, 3 materialized views, configurable TTL retention
## Broadened schemas
Score and feedback tables include full entity hierarchy (entityType,
parentEntityType, rootEntityType), correlation IDs (runId, sessionId,
threadId,
requestId), and deployment context (environment, executionSource,
serviceName) to match the updated core record types from
correlationContext.
## DuckDB parity
All 40 methods match 1:1 with the DuckDB adapter, using
ClickHouse-idiomatic equivalents:
- toStartOfInterval vs time_bucket, quantile() vs percentile_cont(),
has() vs list_contains()
- Parameterized queries via {name:Type} syntax
- Write-time value split (valueString/valueNumber) instead of TRY_CAST
at query time
## Other changes
- Fixed @mastra/core record-builder tests to match broadened
score/feedback record shapes
- Aligned percentile validation (empty-array guard + isFinite check)
with DuckDB
- Cleaned up outdated review/audit files from the design doc directory
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added ClickHouse v-next observability adapter with comprehensive OLAP
analytics.
* Introduced analytics methods for scores and feedback: aggregate,
breakdown, time-series, and percentiles queries.
* Expanded score and feedback schemas with entity hierarchy,
correlation, and environment context fields.
* Added discovery functionality for browsing available data dimensions.
* Integrated configurable retention/TTL support for data lifecycle
management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Abhi Aiyer <abhiaiyer91@gmail.com>
|
History
|
|
|---|---|---|
| .. | ||
| discovery.md | ||
| feedback-events.md | ||
| log-events.md | ||
| metric-events.md | ||
| physical-types.md | ||
| README.md | ||
| score-events.md | ||
| shared.md | ||
| span-events.md | ||
| trace-roots.md | ||
ClickHouse vNext Observability Design
Purpose
Use this file as the entry point for the ClickHouse v-next observability design. Cross-cutting decisions live in the shared doc; table-specific behavior lives in the per-table docs.
Important note:
- this document set is design documentation for the initial
v-nextimplementation - it is intended to guide implementation, not to remain a permanent second source of truth after the implementation and DDL/query code exist
- once
v-nextis implemented, the code and tests should become the authoritative source for ongoing behavior
Design Set
- Shared Design
- Trace Roots
- Span Events
- Metric Events
- Log Events
- Score Events
- Feedback Events
- Discovery Design
- Physical Types
Design Summary
The shared doc captures the cross-cutting v0 decisions, including:
- append-only ClickHouse storage across all five signals, with
ReplacingMergeTreefor tracing andMergeTreefor the other four signals - insert-only tracing with
span_eventsplustrace_roots - tracing retry-idempotency via a tracing-only
dedupeKey - broader shared observability context across logs, metrics, scores, and feedback, with
executionSourceas the execution-context column andscoreSource/feedbackSourceas signal-specific source fields - nullable
traceIdsupport for score and feedback events so those signals can still be recorded outside traces - intentionally narrowed v0 ClickHouse trace-filter semantics: top-level string-only metadata equality via
metadataSearch, and no tracescopefiltering - non-tracing signals intentionally remaining non-idempotent under retries in v0
- refreshable discovery helper tables as a best-effort optional subsystem, not a startup requirement for the core observability adapter
- day-granularity retention
- raw ClickHouse DDL as the source of schema definition
Use the shared doc for the common contract and the per-table docs for physical shape and query behavior.
Scope
- Cloud ClickHouse physical design is the target for this work.
- Mastra runtime should continue writing through the standard storage interface via
DefaultExporter. - ClickHouse
v-nextshould be designed around the batched create path used byDefaultExporter. - Legacy observability methods outside that path are expected to be deprecated and should not drive
v-nextdesign decisions. - Previous DuckDB or other storage implementations may be used as parity references, but they should not be treated as the design source of truth for ClickHouse
v-next. - New code should live under
stores/clickhouse/src/storage/domains/observability/v-next/. - Transition, migration, and cutover planning are intentionally out of scope for this design.
- The existing ClickHouse observability domain can remain separate while
v-nextis implemented.
Rollout Order
- Finalize the shared and per-table docs.
- Implement raw ClickHouse DDL for the five signal tables,
trace_roots,discovery_values,discovery_pairs, and their materialized views. - Implement writes and reads for the five signals.
- Add targeted tests around the risky contract points:
- tracing insert-only routing with ended-span-only persistence
- tracing dedupe behavior for retried
span_events/trace_rootswrites trace_rootspopulation from root-span inserts- discovery helper refresh behavior and staleness expectations
- per-table ordering
- derived trace status semantics
- trace
hasChildError metadataRawvsmetadataSearch- exact filter-surface behavior per signal
- label/tag normalization
- delete eventual consistency
Important note:
- this document set is intentionally about steady-state
v-nextdesign, not transition mechanics - migration, cutover, and coexistence planning should not block v0 implementation work