mirror of
https://github.com/monoscope-tech/monoscope.git
synced 2026-11-16 10:18:01 +00:00
perf(metrics): one INSERT shape for otel_metrics, not ~350
`insertOtelMetrics` built its SQL as `chunksOf 350` rows with one `(…)` tuple
per row IN THE STATEMENT TEXT. Full chunks are all 350, but every call's ragged
tail is a different size, so over time the workload mints ~350 distinct
statement texts — each a distinct prepared statement and a distinct cached plan,
and each carrying 350 x 68 ~= 23,800 bind params that TimeFusion's insert
coercion wraps individually in a CAST. Prod 2026-08-14: TF's plan cache sat at
43% (995 hits / 1311 misses) and pgwire's per-connection prepared-statement
store — still unbounded in COUNT — held the rest.
`bulkInsertOtelLogsAndSpans` already solved this: INSERT … SELECT … FROM
unnest(), one array per column, so the text is independent of batch size. This
makes metrics use the same machinery rather than a second copy of it:
- `OtelCol` becomes `BulkCol r`, polymorphic in the row type, and `column` /
`tsColumn` / `castColumn` / `splitColumn` / `jsonColumn` follow. Spans are
unchanged; `insertUnnestStmt` is now `unnestInsert` applied to its table.
- `unnestInsert` takes the table, its columns and a PG-only suffix (metrics
needs ON CONFLICT DO NOTHING; TimeFusion has no upsert).
- `metricColumns` is 67 `BulkCol MetricRow`, in table order, replacing
`metricRowSql`. `MetricRow` decodes the attribute/resource maps and the native
value columns ONCE per row instead of once per column.
- New combinators for what metrics needs and spans did not: `dateColumn`,
`numArrayColumn` (typed bigint[]/double precision[] rebuilt from 0x1F-joined
text — the same `string_to_array(_, chr(31))::T[]` the per-row form already
emitted on both engines) and `quantileColumn`. Plus `UnnestElem` for `Double`,
`MetricType` and `AggregationTemporality`.
One behaviour change: an EMPTY numeric array now binds NULL rather than
producing `{""}`, which no numeric cast accepts. Histogram points always carry
buckets, so that was latent rather than live.
Verified: column names and ORDER are byte-identical to the previous INSERT
(checked programmatically against the old statement); the gRPC metrics
ingestion test — which reads back from a real TimeFusion — passes; the
integration suite has 13 failures against 15 on unmodified master, and the one
metrics-specific failure reproduces on master in isolation.
This commit is contained in:
parent
59a3f7d34e
commit
d9e58f662b
1 changed files with 189 additions and 149 deletions
Loading…
Add table
Add a link
Reference in a new issue