stt/infra-observability
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a traces dashboard and make the span correlations work

Browse Source 
Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.

The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.

Both span correlations were declared but non-functional.

tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.

tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.

The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-26 07:44:57 +02:00
parent a2f5d725a3
commit 367ebd1da5
4 changed files with 1019 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+45
View File
@@ -213,6 +213,51 @@ Each application dashboard carries the same four sections: an overview of RED
indicators, HTTP traffic, a runtime section specific to the stack in use, and a indicators, HTTP traffic, a runtime section specific to the stack in use, and a
**Logs** row, followed by container resources. **Logs** row, followed by container resources.
### Distributed tracing
`obs-traces` searches traces with TraceQL, assembled from a service selector, a
minimum duration, a status, and a free-text field for conditions the selectors
do not cover. The status and free-text variables hold TraceQL *fragments* rather
than values, so an empty one contributes nothing and the surrounding query stays
valid — "no filter" is the default rather than a special case.
Opening a trace gives the waterfall, every span's attributes and events, and
from a span its logs and its RED metrics. The rest of the dashboard is the
aggregate view: rate, errors and latency percentiles from the span metrics
Tempo's own `metrics_generator` derives from the traces, the slowest and most
failing operations by span name, and the service-to-service edges Tempo infers.
Span name rather than endpoint is the unit worth looking at: a slow database
statement surfaces next to a slow HTTP handler, which is what tracing buys over
plain HTTP metrics.
The interactive node graph stays in Explore — `serviceMap` is a query type
Grafana's frontend assembles and it returns nothing through the dashboard query
API, so the dashboard shows the same edges as a table instead.
#### Span-to-logs correlation
The Tempo datasource resolves a span's logs with a custom query on
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
label mapping it replaces resolved to `app`, which Alloy derives from
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
but not for the six Kong gateways — and the gateways were the only place the
trace id actually appeared. The mapping pointed at the services where it could
not work.
Coverage today, each verified by taking a trace id and searching for it:
| Service | Trace id in logs | How |
|---|---|---|
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
| LibreChat | no | would need the upstream image patched |
`tracesToMetrics` likewise had no queries defined and rendered a bare datasource
link; it now carries rate, error-rate and p95 queries against the span metrics,
scoped to the span's own service.
### Log dashboards, split by kind ### Log dashboards, split by kind
Logs are searched from three dashboards rather than one, because the three Logs are searched from three dashboards rather than one, because the three
k8s/grafana/dashboards/kustomization.yaml
+1
View File
@@ -25,6 +25,7 @@ configMapGenerator:
- obs-kong-gateways.json - obs-kong-gateways.json
- obs-librechat.json - obs-librechat.json
- obs-stack-health.json - obs-stack-health.json
- obs-traces.json
# <<< dashboards <<< # <<< dashboards <<<
generatorOptions: generatorOptions:
k8s/grafana/dashboards/obs-traces.json
+942
View File
@@ -0,0 +1,942 @@
{
"uid": "obs-traces",
"title": "Distributed Traces",
"description": "Search traces with TraceQL, then open one to see its spans, their attributes, and the logs and metrics belonging to each span. RED figures come from the span metrics Tempo derives from the traces themselves.",
"tags": [
"observability",
"traces",
"tempo"
],
"editable": false,
"schemaVersion": 39,
"version": 1,
"refresh": "",
"time": {
"from": "now-1h",
"to": "now"
},
"timezone": "browser",
"templating": {
"list": [
{
"name": "service",
"label": "Service",
"type": "query",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"query": {
"type": 1,
"refId": "service-variable"
},
"refresh": 2,
"sort": 1,
"multi": true,
"includeAll": true,
"allValue": ".+",
"current": {
"text": [
"All"
],
"value": [
"$__all"
]
},
"description": "Service names as Tempo knows them."
},
{
"name": "minDuration",
"label": "Min duration",
"type": "custom",
"query": "0ms,10ms,50ms,100ms,500ms,1s,5s",
"current": {
"text": "0ms",
"value": "0ms"
},
"multi": false,
"includeAll": false
},
{
"name": "status",
"label": "Status",
"type": "custom",
"query": "Any : ,Error only : && status = error,OK only : && status = ok",
"current": {
"text": "Any",
"value": " "
},
"multi": false,
"includeAll": false
},
{
"name": "extra",
"label": "TraceQL",
"type": "textbox",
"query": "",
"current": {
"text": "",
"value": ""
},
"description": "Extra TraceQL conditions, appended inside the braces. Example: && span.http.status_code = 500"
}
]
},
"panels": [
{
"id": 1,
"type": "row",
"title": "Find traces",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"panels": []
},
{
"id": 2,
"type": "table",
"title": "Trace search",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"gridPos": {
"h": 13,
"w": 24,
"x": 0,
"y": 1
},
"description": "TraceQL, assembled from the selectors above. Click a row to open the trace: the waterfall, every span's attributes and events, and — from a span — its logs and its RED metrics, wired on the Tempo datasource. The time range is the dashboard's, so narrowing the picker narrows the search.",
"targets": [
{
"refId": "A",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"queryType": "traceql",
"query": "{ resource.service.name =~ \"${service:regex}\" && duration >= ${minDuration} ${status} ${extra} }",
"limit": 100,
"tableType": "traces"
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "traceIdHidden"
},
"properties": [
{
"id": "custom.hidden",
"value": true
}
]
},
{
"matcher": {
"id": "byName",
"options": "traceDuration"
},
"properties": [
{
"id": "unit",
"value": "ms"
},
{
"id": "custom.width",
"value": 120
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Start time",
"desc": true
}
]
}
},
{
"id": 3,
"type": "row",
"title": "Rate, errors, duration (from span metrics)",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 14
},
"panels": []
},
{
"id": 4,
"type": "stat",
"title": "Spans / s",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 15
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "sum(rate(traces_spanmetrics_calls_total{service=~\"${service:regex}\"}[$__rate_interval]))"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"decimals": 2,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "blue"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
},
"description": "Derived by Tempo's metrics_generator from the traces themselves, then remote-written to Prometheus — no extra instrumentation in the applications."
},
{
"id": 5,
"type": "stat",
"title": "Error rate",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 15
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "(sum(rate(traces_spanmetrics_calls_total{service=~\"${service:regex}\",status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])) or vector(0)) / clamp_min(sum(rate(traces_spanmetrics_calls_total{service=~\"${service:regex}\"}[$__rate_interval])), 0.001)"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"decimals": 2,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.01
},
{
"color": "red",
"value": 0.05
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
},
"description": "Share of spans whose status is STATUS_CODE_ERROR."
},
{
"id": 6,
"type": "stat",
"title": "Latency p95",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 15
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=~\"${service:regex}\"}[$__rate_interval])))"
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"decimals": 0,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 500
},
{
"color": "red",
"value": 2000
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
}
},
{
"id": 7,
"type": "stat",
"title": "Services traced",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 15
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "count(group by (service) (traces_spanmetrics_calls_total{service=~\"${service:regex}\"}))"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "purple"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "purple",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "auto"
}
},
{
"id": 8,
"type": "timeseries",
"title": "Span rate by service",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 19
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "sum by (service) (rate(traces_spanmetrics_calls_total{service=~\"${service:regex}\"}[$__rate_interval]))",
"legendFormat": "{{service}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 8,
"showPoints": "never",
"pointSize": 8,
"gradientMode": "none",
"spanNulls": false,
"axisLabel": "",
"axisPlacement": "auto",
"lineInterpolation": "smooth"
},
"mappings": [],
"color": {
"mode": "palette-classic"
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 9,
"type": "timeseries",
"title": "Latency percentiles",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 19
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "histogram_quantile(0.50, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=~\"${service:regex}\"}[$__rate_interval])))",
"legendFormat": "p50"
},
{
"refId": "B",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "histogram_quantile(0.95, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=~\"${service:regex}\"}[$__rate_interval])))",
"legendFormat": "p95"
},
{
"refId": "C",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "histogram_quantile(0.99, sum by (le) (rate(traces_spanmetrics_latency_bucket{service=~\"${service:regex}\"}[$__rate_interval])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 0,
"showPoints": "never",
"pointSize": 8,
"gradientMode": "none",
"spanNulls": false,
"axisLabel": "",
"axisPlacement": "auto",
"lineInterpolation": "smooth"
},
"mappings": [],
"color": {
"mode": "palette-classic"
},
"min": 0
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "p50"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "light-blue"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "p95"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "blue"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "p99"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "dark-blue"
}
}
]
}
]
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "One hue light to dark: ordered magnitudes of the same measure, not distinct entities."
},
{
"id": 10,
"type": "bargauge",
"title": "Slowest operations (p95)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 27
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "topk(10, histogram_quantile(0.95, sum by (service, span_name, le) (rate(traces_spanmetrics_latency_bucket{service=~\"${service:regex}\"}[$__rate_interval]))))",
"legendFormat": "{{service}} · {{span_name}}",
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "ms",
"min": 0,
"mappings": [],
"color": {
"mode": "continuous-BlPu"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "text",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
},
"description": "Span name, not endpoint: a slow database statement surfaces here next to a slow HTTP handler, which is the point of tracing over plain HTTP metrics."
},
{
"id": 11,
"type": "bargauge",
"title": "Operations in error",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 27
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "topk(10, sum by (service, span_name) (rate(traces_spanmetrics_calls_total{service=~\"${service:regex}\",status_code=\"STATUS_CODE_ERROR\"}[$__rate_interval])))",
"legendFormat": "{{service}} · {{span_name}}",
"instant": true
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0,
"mappings": [],
"color": {
"mode": "continuous-BlPu"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true,
"valueMode": "text",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
}
},
"description": "Empty is the healthy state."
},
{
"id": 12,
"type": "row",
"title": "Service dependencies",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 36
},
"panels": []
},
{
"id": 13,
"type": "table",
"title": "Calls between services",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 37
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "sum by (client, server) (rate(traces_service_graph_request_total[$__rate_interval]))",
"format": "table",
"instant": true
}
],
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true,
"__name__": true
},
"indexByName": {
"client": 0,
"server": 1,
"Value": 2
},
"renameByName": {
"client": "Caller",
"server": "Callee",
"Value": "Requests/s"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Requests/s"
},
"properties": [
{
"id": "unit",
"value": "reqps"
},
{
"id": "decimals",
"value": 3
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Requests/s",
"desc": true
}
]
},
"description": "Edges Tempo infers from the traces themselves — no service registry involved. `user` as caller means the span had no parent, so the call entered the system there. The interactive node graph lives in Explore, on the Tempo datasource's Service Graph tab."
},
{
"id": 14,
"type": "timeseries",
"title": "Latency between services",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 37
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"expr": "histogram_quantile(0.95, sum by (client, server, le) (rate(traces_service_graph_request_server_seconds_bucket[$__rate_interval])))",
"legendFormat": "{{client}} → {{server}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 0,
"showPoints": "never",
"pointSize": 8,
"gradientMode": "none",
"spanNulls": false,
"axisLabel": "",
"axisPlacement": "auto",
"lineInterpolation": "smooth"
},
"mappings": [],
"color": {
"mode": "palette-classic"
},
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "p95 as measured on the server side of each edge."
}
]
}
k8s/grafana/provisioning/datasources.yaml
+31 -8
View File
@@ -64,19 +64,42 @@ datasources:
datasourceUid: prometheus datasourceUid: prometheus
nodeGraph: nodeGraph:
enabled: true enabled: true
# Span -> logs by trace id rather than by service name.
#
# The service-name mapping this replaces resolved `service.name` to Loki's
# `app` label, which Alloy derives from app.kubernetes.io/name. That label
# exists for keycloak and classic-models-api but not for the six Kong
# gateways or librechat — and the gateways are precisely where the trace
# id does appear in the logs. The mapping pointed at the services where it
# could not work.
#
# A custom query on ${__trace.traceId} sidesteps the label entirely: every
# source that writes the id anywhere in the line is matched. The stream
# selector is spelled out rather than `.+` so the query stays bounded.
tracesToLogsV2: tracesToLogsV2:
datasourceUid: loki datasourceUid: loki
spanStartTimeShift: -5m spanStartTimeShift: -1h
spanEndTimeShift: 5m spanEndTimeShift: 1h
filterByTraceID: true filterByTraceID: false
filterBySpanID: false filterBySpanID: false
tags: customQuery: true
- key: service.name query: '{job=~"kong-audit|keycloak/keycloak|classic-models/classic-models-api|librechat/librechat"} |= "$${__trace.traceId}"'
value: app # Span -> metrics against the span metrics Tempo's own metrics_generator
# feeds into Prometheus. Without explicit queries this block renders a
# bare datasource link and nothing else, which is what it did before.
# $$__tags expands to the tag mapping below, so each query is scoped to
# the span's own service.
tracesToMetrics: tracesToMetrics:
datasourceUid: prometheus datasourceUid: prometheus
spanStartTimeShift: -5m spanStartTimeShift: -30m
spanEndTimeShift: 5m spanEndTimeShift: 30m
tags: tags:
- key: service.name - key: service.name
value: service value: service
queries:
- name: 'Request rate'
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags}[$$__rate_interval]))'
- name: 'Error rate'
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags,status_code="STATUS_CODE_ERROR"}[$$__rate_interval]))'
- name: 'Latency p95'
query: 'histogram_quantile(0.95, sum by (span_name, le) (rate(traces_spanmetrics_latency_bucket{$$__tags}[$$__rate_interval])))'