Compare commits
9 Commits
940abdf6fc
...
c95af8209e
| Author | SHA1 | Date | |
|---|---|---|---|
 Stéphane Tailland
|
c95af8209e | ||
|
Stéphane Tailland
|
d3abde1040 | ||
|
Stéphane Tailland
|
367ebd1da5 | ||
|
Stéphane Tailland
|
a2f5d725a3 | ||
|
Stéphane Tailland
|
7c416ce06f | ||
|
Stéphane Tailland
|
eb35419ad0 | ||
|
Stéphane Tailland
|
40ffac4acf | ||
|
Stéphane Tailland
|
7a8a476ef0 | ||
|
Stéphane Tailland
|
0ba38bc8f7 |
@@ -46,7 +46,8 @@ help: ## Show this help
|
||||
|
||||
.PHONY: deploy
|
||||
deploy: check-context secret ## Deploy the whole stack (runtimes + dashboards)
|
||||
$(KUBECTL) apply -k $(K8S)
|
||||
# --server-side for the same reason as the dashboards target below.
|
||||
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts
|
||||
$(MAKE) --no-print-directory wait
|
||||
|
||||
.PHONY: deploy-runtime
|
||||
@@ -89,7 +90,11 @@ secret: ## Generate k8s/secret.yaml (random admin password) if absent
|
||||
|
||||
.PHONY: dashboards
|
||||
dashboards: check-context dashboards-check ## Apply dashboards only (no Grafana restart)
|
||||
$(KUBECTL) apply -k $(DASHBOARDS)
|
||||
# Server-side apply: client-side apply stores a full copy of the object in the
|
||||
# kubectl.kubernetes.io/last-applied-configuration annotation, which is capped at
|
||||
# 256KB. The dashboards passed that in aggregate. Server-side apply tracks field
|
||||
# ownership in metadata instead, so the ceiling becomes etcd's ~1MB per object.
|
||||
$(KUBECTL) apply -k $(DASHBOARDS) --server-side --force-conflicts
|
||||
printf "$(C_OK)grafana-dashboards ConfigMap updated$(C_OFF) -- visible in Grafana within ~70s\n"
|
||||
printf " (kubelet volume sync ~60s + provisioner poll 10s, no restart).\n"
|
||||
printf " For immediate effect: $(C_WARN)make restart-grafana$(C_OFF)\n"
|
||||
@@ -310,7 +315,7 @@ _flush:
|
||||
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=0
|
||||
$(KUBECTL) -n $(NS) wait --for=delete pod -l app=$(COMP) --timeout=120s 2>/dev/null || true
|
||||
$(KUBECTL) -n $(NS) delete pvc $(COMP)-data --ignore-not-found --wait=true
|
||||
$(KUBECTL) apply -k $(K8S)/$(COMP)
|
||||
$(KUBECTL) apply -k $(K8S)/$(COMP) --server-side --force-conflicts
|
||||
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=1
|
||||
|
||||
##@ Validation and teardown
|
||||
@@ -326,7 +331,7 @@ diff: check-context ## Diff the manifests against the cluster state
|
||||
.PHONY: validate
|
||||
validate: dashboards-check ## Validate the manifests (build + server dry-run)
|
||||
kustomize build $(K8S) >/dev/null && printf "$(C_OK)kustomize build OK$(C_OFF)\n"
|
||||
$(KUBECTL) apply -k $(K8S) --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
|
||||
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
|
||||
|
||||
.PHONY: undeploy
|
||||
undeploy: ## DESTRUCTIVE: delete the stack, PVCs kept (CONFIRM=yes)
|
||||
|
||||
@@ -132,6 +132,13 @@ forces a pod restart when they change.
|
||||
| `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
|
||||
| `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |
|
||||
|
||||
> **Applies are server-side.** Client-side `kubectl apply` stores a full copy of
|
||||
> each object in the `kubectl.kubernetes.io/last-applied-configuration`
|
||||
> annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate
|
||||
> at eight dashboards. `--server-side` tracks field ownership in metadata
|
||||
> instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one
|
||||
> ConfigMap per dashboard is the next step if that is ever reached.
|
||||
|
||||
StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
|
||||
use `strategy: Recreate`: with an RWO volume the old pod must release it before
|
||||
the new one can start.
|
||||
@@ -191,6 +198,14 @@ changes in `container_start_time_seconds`.
|
||||
| `librechat` | OTel SDK → `tempo:4318` | built-in `/metrics`, Bearer token | stdout | `obs-librechat` |
|
||||
| `classic-models-api` | `opentelemetry-instrument` → `tempo:4318` | django-prometheus `/metrics` | JSON + `otelTraceID` | `obs-classic-models-api` |
|
||||
|
||||
The six Kong gateways share a single dashboard, `obs-kong-gateways`, with a
|
||||
`gateway` selector at the top (one, several, or all). Its variable resolves a
|
||||
subtlety of this setup: the metrics label is the full gateway name
|
||||
(`catalog-dev`) while the audit-log label is the short domain (`catalog`). The
|
||||
variable holds the short form, PromQL re-adds the `-dev` suffix, and LogQL uses
|
||||
it as-is — so one control drives both signals. The capture group in its regex
|
||||
also drops the unrelated `kong` gateway, which carries no `-dev` suffix.
|
||||
|
||||
Their manifests live in the `infra-keycloak`, `infra-librechat` and
|
||||
`classic-models-api` repositories; this repository only holds the platform.
|
||||
|
||||
@@ -198,6 +213,135 @@ Each application dashboard carries the same four sections: an overview of RED
|
||||
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
|
||||
**Logs** row, followed by container resources.
|
||||
|
||||
### Distributed tracing
|
||||
|
||||
Traces are searched and explored in **Explore**, on the Tempo datasource, not
|
||||
from a dashboard. Explore's TraceQL query builder, autocompletion, trace view
|
||||
and interactive service graph have no dashboard equivalent worth maintaining —
|
||||
a dashboard search panel is a worse copy of a tool that already exists.
|
||||
|
||||
What the repository does own is the datasource wiring, which applies in Explore
|
||||
exactly as it would in a dashboard.
|
||||
|
||||
#### Span-to-logs correlation
|
||||
|
||||
The Tempo datasource resolves a span's logs with a custom query on
|
||||
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
|
||||
label mapping it replaces resolved to `app`, which Alloy derives from
|
||||
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
|
||||
but not for the six Kong gateways — and the gateways were the only place the
|
||||
trace id actually appeared. The mapping pointed at the services where it could
|
||||
not work.
|
||||
|
||||
Coverage today, each verified by taking a trace id and searching for it:
|
||||
|
||||
| Service | Trace id in logs | How |
|
||||
|---|---|---|
|
||||
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
|
||||
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
|
||||
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
|
||||
| LibreChat | no | would need the upstream image patched |
|
||||
|
||||
#### Span-to-metrics correlation
|
||||
|
||||
`tracesToMetrics` had no queries defined and rendered a bare datasource link. It
|
||||
now carries rate, error-rate and p95 queries against `traces_spanmetrics_*`,
|
||||
scoped to the span's own service.
|
||||
|
||||
Those span metrics are worth knowing about independently: Tempo's
|
||||
`metrics_generator` derives them from the traces and remote-writes them to
|
||||
Prometheus, keyed by `service` and `span_name`. They measure the *span*, so a
|
||||
slow database statement appears next to a slow HTTP handler — a granularity the
|
||||
gateway and per-application dashboards, which measure HTTP, do not have.
|
||||
|
||||
### Log dashboards, split by kind
|
||||
|
||||
Logs are searched from three dashboards rather than one, because the three
|
||||
questions have nothing in common and a single view compromised on all of them.
|
||||
|
||||
| Dashboard | Question | Sources |
|
||||
|---|---|---|
|
||||
| `obs-access-logs` | Which HTTP requests came in? | Kong records, Keycloak access logger, Django gunicorn, ingress-nginx |
|
||||
| `obs-application-logs` | What is the code reporting, and what broke? | Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log |
|
||||
| `obs-audit-logs` | Who authenticated, what changed, who called what? | Keycloak `org.keycloak.events`, Kong http-log records |
|
||||
|
||||
Each source keeps its own parser and they all emit the same field names, so the
|
||||
frames merge into one table with consistent columns.
|
||||
|
||||
Keycloak's identity events are read with one independent `regexp` per field
|
||||
rather than `line_format` piped into `logfmt`. Two reasons, both measured: that
|
||||
combination is pathological on this data — each stage answers in 0.3s alone,
|
||||
chained they time out at 60s and the table never renders — and the field set
|
||||
differs per event type, so a single regexp spanning them would fail whole rows
|
||||
on a missing field. Independent stages leave the absent field empty and keep the
|
||||
row, and the query returns in 0.1s. Access logs converge on
|
||||
method / path / status / client; application logs on level / message.
|
||||
|
||||
Three things this split fixes that a single consolidated view could not:
|
||||
|
||||
**The kinds are subtracted, not just filtered.** Application logs exclude
|
||||
Keycloak's `access-log` and `events` loggers by name, the Django container's
|
||||
NCSA lines by shape, and the Kong containers' stray access lines by requiring
|
||||
the nginx date prefix. Without that, access lines drown the signal.
|
||||
|
||||
**Levels are read from the line, not inferred.** Loki's `detected_level` marks
|
||||
78% of the Kong gateway output `unknown` even though every one of those lines
|
||||
states `[notice]` or `[warn]`. Each parser extracts the level and lowercases it,
|
||||
so nginx's `notice`, Keycloak's `INFO` and LibreChat's `info` land on one scale
|
||||
— and the level selector has no `unknown` entry, because a line reaching the
|
||||
table always states one.
|
||||
|
||||
**The scope selector actually scopes.** Each query carries both
|
||||
`job=~"${source:regex}"` and a literal job matcher. With a single selector,
|
||||
choosing "All" widened the query back to every stream in the cluster — which is
|
||||
how MongoDB and cert-manager ended up in an application view.
|
||||
|
||||
Latency is populated for Kong only: the NCSA format the other sources use
|
||||
carries no timing field. That is a property of the format, not a gap in the
|
||||
query.
|
||||
|
||||
The NCSA sources are read with `regexp`, not `pattern`. A trailing `pattern`
|
||||
capture consumes the rest of the line, so a `<size>` at the end swallowed the
|
||||
referer and user-agent along with the byte count. Bounding the capture on `\d+`
|
||||
stops it at the number, and `regexp` tolerates both shapes — Keycloak's access
|
||||
line ends at the size, gunicorn's and nginx's carry more fields after it.
|
||||
|
||||
In every table the raw record is the last column, and the `organize`
|
||||
transformation's `indexByName` keys on the original field names rather than the
|
||||
renamed labels — keying it on the display names silently leaves the column order
|
||||
to whatever order the frames arrived in.
|
||||
|
||||
### Parsed log tables### Parsed log tables
|
||||
|
||||
Every dashboard shows its logs twice: a **table** with one column per parsed
|
||||
field, and, in a collapsed row underneath, the original **logs panel**.
|
||||
|
||||
The table is what makes a JSON record readable — status, method, route, latency
|
||||
as sortable, filterable columns rather than a wall of braces. Its last column
|
||||
holds the unparsed record, and every cell has the inspector enabled, so the raw
|
||||
line is one click away.
|
||||
|
||||
The logs panel stays because a table cannot do what it does: render ANSI colour,
|
||||
expand a record inline, tail live, and turn a `trace_id` into a link to Tempo.
|
||||
It is collapsed so it costs nothing until needed.
|
||||
|
||||
Parsing is per-format, because these five sources agree on nothing:
|
||||
|
||||
| Source | Format | Parser |
|
||||
|---|---|---|
|
||||
| Kong audit | nested JSON | `\| json` with explicit field paths |
|
||||
| Keycloak | JSON | `\| json` |
|
||||
| classic-models-api | NCSA access lines | `\| pattern` |
|
||||
| LibreChat | ANSI-coloured text | `line_format` to strip ANSI, then `\| pattern` |
|
||||
| Observability stack | logfmt | `\| logfmt` with named fields |
|
||||
|
||||
Two choices worth knowing. The Kong and stack parsers name their fields
|
||||
explicitly rather than extracting everything: a bare `\| json` on a Kong record
|
||||
yields about seventy columns, and a bare `\| logfmt` explodes Loki's own
|
||||
query-metrics lines. And `\| pattern` keeps lines it cannot match, with empty
|
||||
columns, rather than dropping them — so the Django table never silently hides
|
||||
an application log among the access lines.
|
||||
|
||||
The Logs row leans on Loki's `detected_level`, which is inferred at ingestion
|
||||
time, so a single query works across all three log formats — Keycloak emits JSON,
|
||||
LibreChat colour-coded text, and the Django API plain gunicorn access lines. No
|
||||
|
||||
@@ -17,8 +17,13 @@ configMapGenerator:
|
||||
- name: grafana-dashboards
|
||||
files:
|
||||
# >>> dashboards (generated by `make dashboards-sync`) >>>
|
||||
- obs-access-logs.json
|
||||
- obs-application-logs.json
|
||||
- obs-audit-logs.json
|
||||
- obs-classic-models-api.json
|
||||
- obs-distributed-traces.json
|
||||
- obs-keycloak.json
|
||||
- obs-kong-gateways.json
|
||||
- obs-librechat.json
|
||||
- obs-stack-health.json
|
||||
# <<< dashboards <<<
|
||||
|
||||
@@ -0,0 +1,846 @@
|
||||
{
|
||||
"uid": "obs-access-logs",
|
||||
"title": "Access Logs",
|
||||
"description": "One row per HTTP request, across the Kong gateways, Keycloak, the Classic Models API and the ingress controller. Each source is parsed by its own format and normalised to the same columns.",
|
||||
"tags": [
|
||||
"observability",
|
||||
"logs",
|
||||
"access"
|
||||
],
|
||||
"editable": false,
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "source",
|
||||
"label": "Source",
|
||||
"type": "custom",
|
||||
"query": "Kong gateways : kong-audit,Keycloak : keycloak/keycloak,Classic Models API : classic-models/classic-models-api,Ingress NGINX : ingress-nginx/controller",
|
||||
"multi": true,
|
||||
"includeAll": true,
|
||||
"allValue": "kong-audit|keycloak/keycloak|classic-models/classic-models-api|ingress-nginx/controller",
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "search",
|
||||
"label": "Search",
|
||||
"type": "textbox",
|
||||
"query": "",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"description": "Case-insensitive regex over the whole line. Empty matches everything."
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Requests",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum(count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range])) or vector(0)",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "blue"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "blue",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
},
|
||||
"description": "Every line the selected sources emitted over the dashboard's time range."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Client errors (4xx)",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | status =~ \"4..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"4..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"4..\" [$__range])) or vector(0))",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
},
|
||||
"description": "Summed across all four sources, each parsed by its own format."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "stat",
|
||||
"title": "Server errors (5xx)",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | status =~ \"5..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"5..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"5..\" [$__range])) or vector(0))",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Sources active",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "count(sum by (job) (count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range]))) or vector(0)",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "purple"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "purple",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Requests by source",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 80,
|
||||
"showPoints": "never",
|
||||
"barAlignment": 0,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"showLegend": true,
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": [
|
||||
"sum"
|
||||
]
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"description": "Bars rather than lines: a request count per bucket is a run of discrete events."
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Status classes",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
|
||||
"legendFormat": "{{class}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
|
||||
"legendFormat": "{{class}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
|
||||
"legendFormat": "{{class}}",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 80,
|
||||
"showPoints": "never",
|
||||
"barAlignment": 0,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "2xx"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "green"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "3xx"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "blue"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "4xx"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "orange"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "5xx"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "red"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"showLegend": true,
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": [
|
||||
"sum"
|
||||
]
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"description": "Derived from each source's parsed status, so the formats land on one scale. A class can appear once per query in the legend; the stack itself is the correct total."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "table",
|
||||
"title": "Requests",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 14,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
}
|
||||
],
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "merge",
|
||||
"options": {}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|status|method|path|job|client|latency|size)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"status": 1,
|
||||
"method": 2,
|
||||
"path": 3,
|
||||
"job": 4,
|
||||
"client": 5,
|
||||
"latency": 6,
|
||||
"size": 7,
|
||||
"body": 8
|
||||
},
|
||||
"renameByName": {
|
||||
"timestamp": "Time",
|
||||
"body": "Raw line",
|
||||
"status": "Status",
|
||||
"method": "Method",
|
||||
"path": "Path",
|
||||
"job": "Source",
|
||||
"client": "Client",
|
||||
"latency": "Latency (ms)",
|
||||
"size": "Bytes"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Status"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 80
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^2..$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^3..$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^4..$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^5..$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Method"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Source"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 230
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Client"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 130
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Latency (ms)"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 110
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Bytes"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw line"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 240
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "One row per request across every source. Latency is populated for Kong only — the NCSA format the others use carries no timing field, which is a property of the format rather than a gap in the query. Raw line holds the record as emitted."
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "logs",
|
||||
"title": "Requests as a log stream",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\"",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": true,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Same result set. Kept for live tail and for the trace link the Loki datasource derives from a trace id, neither of which a table can do."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,926 @@
|
||||
{
|
||||
"uid": "obs-application-logs",
|
||||
"title": "Application Logs",
|
||||
"description": "What the applications themselves report, with access lines and audit events subtracted. Levels are read from the line rather than inferred. Defaults to errors and warnings.",
|
||||
"tags": [
|
||||
"observability",
|
||||
"logs",
|
||||
"application"
|
||||
],
|
||||
"editable": false,
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "source",
|
||||
"label": "Source",
|
||||
"type": "custom",
|
||||
"query": "Keycloak : keycloak/keycloak,LibreChat : librechat/librechat,Classic Models API : classic-models/classic-models-api,Kong gateways : (catalog|customers|payments|sales|workforce|llm)/proxy",
|
||||
"multi": true,
|
||||
"includeAll": true,
|
||||
"allValue": "keycloak/keycloak|librechat/librechat|classic-models/classic-models-api|(catalog|customers|payments|sales|workforce|llm)/proxy",
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "level",
|
||||
"label": "Level",
|
||||
"type": "custom",
|
||||
"query": "error,warn,notice,info,debug,crit",
|
||||
"multi": true,
|
||||
"includeAll": true,
|
||||
"allValue": "error|warn|notice|info|debug|crit|critical|fatal|emerg|alert",
|
||||
"current": {
|
||||
"text": [
|
||||
"error",
|
||||
"warn"
|
||||
],
|
||||
"value": [
|
||||
"error",
|
||||
"warn"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "search",
|
||||
"label": "Search",
|
||||
"type": "textbox",
|
||||
"query": "",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"description": "Case-insensitive regex over the whole line. Empty matches everything."
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Errors",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0))",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
},
|
||||
"description": "Independent of the Level selector, so it still answers \"is anything broken here?\" while you are filtered on something else."
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Warnings",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0))",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "text",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "orange",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "stat",
|
||||
"title": "Lines matching",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0))",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "blue"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "blue",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Sources active",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "count(sum by (job) (count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range]))) or vector(0)",
|
||||
"queryType": "instant"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"decimals": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "purple"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "purple",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"textMode": "auto"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Volume by level",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 4
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 80,
|
||||
"showPoints": "never",
|
||||
"barAlignment": 0,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "critical"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "dark-red"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "error"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "red"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "warn"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "orange"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "info"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "green"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "debug"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "blue"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "unknown"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "notice"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "text"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"showLegend": true,
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": [
|
||||
"sum"
|
||||
]
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"description": "Levels as each runtime states them, lowercased so nginx's `notice` and Keycloak's `INFO` land on the same scale."
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Errors by source",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 4
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
|
||||
"legendFormat": "{{job}}",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0,
|
||||
"mappings": [],
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"drawStyle": "bars",
|
||||
"lineWidth": 1,
|
||||
"fillOpacity": 80,
|
||||
"showPoints": "never",
|
||||
"barAlignment": 0,
|
||||
"stacking": {
|
||||
"mode": "normal",
|
||||
"group": "A"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"showLegend": true,
|
||||
"displayMode": "table",
|
||||
"placement": "bottom",
|
||||
"calcs": [
|
||||
"sum"
|
||||
]
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"description": "Empty is the healthy state."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "table",
|
||||
"title": "Application log lines",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 14,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 12
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 1000
|
||||
}
|
||||
],
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "merge",
|
||||
"options": {}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|job|level|msg|logger)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"job": 1,
|
||||
"level": 2,
|
||||
"msg": 3,
|
||||
"logger": 4,
|
||||
"body": 5
|
||||
},
|
||||
"renameByName": {
|
||||
"timestamp": "Time",
|
||||
"body": "Raw line",
|
||||
"job": "Source",
|
||||
"level": "Level",
|
||||
"msg": "Message",
|
||||
"logger": "Logger"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Source"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 210
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Level"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(fatal|FATAL|critical|CRITICAL)$",
|
||||
"result": {
|
||||
"color": "dark-red",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(error|ERROR|Error)$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(warn|WARN|warning|WARNING)$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(info|INFO|Info)$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(debug|DEBUG|Debug|trace|TRACE)$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 4
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^unknown$",
|
||||
"result": {
|
||||
"color": "text",
|
||||
"index": 5
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Logger"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 240
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw line"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 240
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Each runtime parsed by its own format — Keycloak JSON, LibreChat colour-coded text, Django JSON, Kong's nginx error log — normalised to level and message. Logger is populated for Keycloak only, the one source that names the emitting class."
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "logs",
|
||||
"title": "Application logs as a stream",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range"
|
||||
},
|
||||
{
|
||||
"refId": "D",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": true,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Same result set, with ANSI colour and inline detail expansion."
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1384,9 +1384,9 @@
|
||||
"description": "Namespace-wide so the datastore's failures show up next to the application's."
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"type": "logs",
|
||||
"title": "Recent errors and warnings",
|
||||
"id": 25,
|
||||
"type": "table",
|
||||
"title": "Access log",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
@@ -1404,21 +1404,274 @@
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"classic-models\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
"expr": "{namespace=\"classic-models\", container=\"classic-models-api\"} | pattern `<client> - - [<ts>] \"<method> <uri> <proto>\" <status> <size> <_> \"<agent>\"`",
|
||||
"queryType": "range",
|
||||
"maxLines": 500
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|status|method|uri|size|client|agent)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"status": 1,
|
||||
"method": 2,
|
||||
"uri": 3,
|
||||
"size": 4,
|
||||
"client": 5,
|
||||
"agent": 6,
|
||||
"body": 7
|
||||
},
|
||||
"renameByName": {
|
||||
"status": "Status",
|
||||
"method": "Method",
|
||||
"uri": "URI",
|
||||
"size": "Bytes",
|
||||
"client": "Client",
|
||||
"agent": "User agent",
|
||||
"body": "Raw record",
|
||||
"timestamp": "Time"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Status"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 80
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^2..$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^3..$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^4..$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^5..$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Method"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "URI"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 300
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Bytes"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Client"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 130
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "User agent"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 200
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw record"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 260
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "gunicorn writes NCSA access lines, not JSON, so these are split by pattern. Lines that do not match — application JSON logs — are kept with empty columns rather than dropped."
|
||||
},
|
||||
{
|
||||
"id": 26,
|
||||
"type": "row",
|
||||
"title": "Raw log lines",
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 58
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 20,
|
||||
"type": "logs",
|
||||
"title": "Raw log lines",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 11,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 59
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"classic-models\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
@@ -1429,7 +1682,7 @@
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 58
|
||||
"y": 59
|
||||
},
|
||||
"panels": []
|
||||
},
|
||||
@@ -1445,7 +1698,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 59
|
||||
"y": 60
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1511,7 +1764,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 59
|
||||
"y": 60
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1576,7 +1829,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 59
|
||||
"y": 60
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
{
|
||||
"uid": "obs-distributed-traces",
|
||||
"title": "Distributed Traces",
|
||||
"description": "Find a trace, and keep only the spans worth looking at. Built because Explore's unfiltered search is unusable here: Tempo does not return {} newest-first, so health checks crowd out real transactions.\n\nDeliberately carries no metric panels. A span's own duration is in the span tables, measured; aggregate RED by span name is what the per-application dashboards already show, and neither answers 'the metrics of this span'.",
|
||||
"tags": [
|
||||
"observability",
|
||||
"tracing",
|
||||
"tempo"
|
||||
],
|
||||
"timezone": "browser",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "",
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"editable": true,
|
||||
"graphTooltip": 1,
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "service",
|
||||
"label": "Service",
|
||||
"type": "query",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"query": {
|
||||
"query": "label_values(traces_spanmetrics_calls_total, service)",
|
||||
"refId": "service"
|
||||
},
|
||||
"refresh": 2,
|
||||
"multi": true,
|
||||
"includeAll": true,
|
||||
"allValue": ".+",
|
||||
"current": {
|
||||
"text": [
|
||||
"All"
|
||||
],
|
||||
"value": [
|
||||
"$__all"
|
||||
]
|
||||
},
|
||||
"description": "Services known to Tempo's span metrics. Drives the trace, span and metric panels."
|
||||
},
|
||||
{
|
||||
"name": "traceId",
|
||||
"label": "Trace ID",
|
||||
"type": "textbox",
|
||||
"query": "",
|
||||
"current": {
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"description": "Set by clicking a Trace ID in the top table, or paste one. Empty means the log filter matches everything."
|
||||
}
|
||||
]
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Traces",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "Traces on the dashboard time range",
|
||||
"description": "TraceQL over the dashboard's time range. Filtered by service so the health-check noise stays out: an unfiltered {} query is NOT returned newest-first by Tempo and the limit truncates arbitrarily, which is why browsing raw Explore misses recent traces. Click a Trace ID to load it below.",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 9,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"queryType": "traceql",
|
||||
"query": "{resource.service.name=~\"$service\"}",
|
||||
"tableType": "traces",
|
||||
"limit": 50,
|
||||
"spss": 20
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"filterable": true
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "traceIdHidden"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.hidden",
|
||||
"value": true
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "traceID"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "links",
|
||||
"value": [
|
||||
{
|
||||
"title": "Load this trace below",
|
||||
"url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}&var-service=${service:queryparam}"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Start time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Spans that matter",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 10
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "HTTP hops between components",
|
||||
"description": "One row per component entry: these are the SERVER-kind spans, i.e. the inbound HTTP request each service received. Browser to LibreChat, LibreChat to the llm gateway, gateway to a domain gateway, gateway to the backend. Everything else — express middleware, DNS, SQL — is deliberately excluded.",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 11
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"queryType": "traceql",
|
||||
"query": "{resource.service.name=~\"$service\" && kind=server} | select(span.http.method, span.http.status_code, span.http.route, span.http.url)",
|
||||
"tableType": "spans",
|
||||
"limit": 50,
|
||||
"spss": 20
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"filterable": true
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "table",
|
||||
"title": "Kong plugin execution",
|
||||
"description": "Every plugin Kong ran, with its own duration — openid-connect, rate-limiting, request-transformer, opentelemetry, and so on. This is what shows which plugin is costing latency on a gateway hop.",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 11
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "tempo",
|
||||
"uid": "tempo"
|
||||
},
|
||||
"queryType": "traceql",
|
||||
"query": "{resource.service.name=~\"$service\" && name=~\"kong\\\\..*plugin.*\"}",
|
||||
"tableType": "spans",
|
||||
"limit": 50,
|
||||
"spss": 20
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"filterable": true
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"showHeader": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "row",
|
||||
"title": "Logs for the selected trace",
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 21
|
||||
},
|
||||
"collapsed": false,
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"type": "logs",
|
||||
"title": "Logs carrying trace $traceId",
|
||||
"description": "Correlation is by TRACE id, not span id: none of the emitters log a span id, so this is every log line of the whole trace rather than of one span.\n\nCoverage is uneven and worth knowing: kong-audit and the backend's access log carry the id on every line; the backend's application log does too since v1.0.4. LibreChat logs carry no trace id at all, so its lines never appear here.\n\nWith traceId empty the filter matches everything — paste an id or click one above.",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 11,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 22
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{job=~\"kong-audit|classic-models/classic-models-api|librechat/librechat|keycloak/keycloak\"} |= \"$traceId\""
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Ascending",
|
||||
"enableLogDetails": true
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1387,9 +1387,9 @@
|
||||
"description": "Namespace-wide so the datastore's failures show up next to the application's."
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"type": "logs",
|
||||
"title": "Recent errors and warnings",
|
||||
"id": 26,
|
||||
"type": "table",
|
||||
"title": "Errors and warnings",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
@@ -1407,21 +1407,244 @@
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"keycloak\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
"expr": "{namespace=\"keycloak\", container=\"keycloak\"} | json | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 500
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|level|loggerName|message|threadName)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"level": 1,
|
||||
"loggerName": 2,
|
||||
"message": 3,
|
||||
"threadName": 4,
|
||||
"body": 5
|
||||
},
|
||||
"renameByName": {
|
||||
"level": "Level",
|
||||
"loggerName": "Logger",
|
||||
"message": "Message",
|
||||
"threadName": "Thread",
|
||||
"body": "Raw record",
|
||||
"timestamp": "Time"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Level"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
|
||||
"result": {
|
||||
"color": "dark-red",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(error|ERROR|Error)$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(info|INFO|Info)$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(debug|DEBUG|Debug)$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Logger"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 280
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Thread"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 160
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw record"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 260
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Keycloak logs JSON, so the fields come straight from `| json`. Scoped to the Keycloak container: PostgreSQL in the same namespace is not JSON and would parse to empty columns."
|
||||
},
|
||||
{
|
||||
"id": 27,
|
||||
"type": "row",
|
||||
"title": "Raw log lines",
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 60
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 21,
|
||||
"type": "logs",
|
||||
"title": "Raw log lines",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 11,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 61
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"keycloak\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
@@ -1432,7 +1655,7 @@
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 60
|
||||
"y": 61
|
||||
},
|
||||
"panels": []
|
||||
},
|
||||
@@ -1448,7 +1671,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1514,7 +1737,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1579,7 +1802,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1522,9 +1522,9 @@
|
||||
"description": "Namespace-wide so the datastore's failures show up next to the application's."
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"type": "logs",
|
||||
"title": "Recent errors and warnings",
|
||||
"id": 27,
|
||||
"type": "table",
|
||||
"title": "Errors and warnings",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
@@ -1542,21 +1542,216 @@
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"librechat\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
"expr": "{namespace=\"librechat\", container=\"librechat\"} | detected_level =~ \"error|critical|warn\" | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<date> <time> <lvl>: <msg>`",
|
||||
"queryType": "range",
|
||||
"maxLines": 500
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|lvl|msg)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"lvl": 1,
|
||||
"msg": 2,
|
||||
"body": 3
|
||||
},
|
||||
"renameByName": {
|
||||
"lvl": "Level",
|
||||
"msg": "Message",
|
||||
"body": "Raw record",
|
||||
"timestamp": "Time"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Level"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
|
||||
"result": {
|
||||
"color": "dark-red",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(error|ERROR|Error)$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(info|INFO|Info)$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(debug|DEBUG|Debug)$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw record"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 260
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "LibreChat writes colour-coded text rather than JSON, so the ANSI escapes are stripped first and the remainder split by pattern. The Raw column therefore holds the cleaned line; the collapsed panel below keeps the original with its colours."
|
||||
},
|
||||
{
|
||||
"id": 28,
|
||||
"type": "row",
|
||||
"title": "Raw log lines",
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 60
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 22,
|
||||
"type": "logs",
|
||||
"title": "Raw log lines",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 11,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 61
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"librechat\"} | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"showCommonLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"prettifyLogMessage": false,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
@@ -1567,7 +1762,7 @@
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 60
|
||||
"y": 61
|
||||
},
|
||||
"panels": []
|
||||
},
|
||||
@@ -1583,7 +1778,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1649,7 +1844,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
@@ -1714,7 +1909,7 @@
|
||||
"h": 7,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 61
|
||||
"y": 62
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
|
||||
@@ -400,9 +400,9 @@
|
||||
"panels": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "logs",
|
||||
"title": "Recent stack errors",
|
||||
"id": 11,
|
||||
"type": "table",
|
||||
"title": "Stack errors and warnings",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
@@ -420,18 +420,256 @@
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"observability\"} |~ \"(?i)(error|level=error|fatal)\"",
|
||||
"queryType": "range"
|
||||
"expr": "{namespace=\"observability\"} | logfmt level, msg, caller, logger, component_id | detected_level =~ \"error|critical|warn\"",
|
||||
"queryType": "range",
|
||||
"maxLines": 500
|
||||
}
|
||||
],
|
||||
"transformations": [
|
||||
{
|
||||
"id": "extractFields",
|
||||
"options": {
|
||||
"source": "labels",
|
||||
"replace": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "filterFieldsByName",
|
||||
"options": {
|
||||
"include": {
|
||||
"pattern": "^(timestamp|body|container|level|msg|caller|component_id)$"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "organize",
|
||||
"options": {
|
||||
"excludeByName": {},
|
||||
"indexByName": {
|
||||
"timestamp": 0,
|
||||
"container": 1,
|
||||
"level": 2,
|
||||
"msg": 3,
|
||||
"caller": 4,
|
||||
"component_id": 5,
|
||||
"body": 6
|
||||
},
|
||||
"renameByName": {
|
||||
"container": "Component",
|
||||
"level": "Level",
|
||||
"msg": "Message",
|
||||
"caller": "Caller",
|
||||
"component_id": "Component id",
|
||||
"body": "Raw record",
|
||||
"timestamp": "Time"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": {
|
||||
"type": "auto"
|
||||
},
|
||||
"inspect": true,
|
||||
"filterable": true
|
||||
},
|
||||
"mappings": []
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Component"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 120
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Level"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 90
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
|
||||
"result": {
|
||||
"color": "dark-red",
|
||||
"index": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(error|ERROR|Error)$",
|
||||
"result": {
|
||||
"color": "red",
|
||||
"index": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
|
||||
"result": {
|
||||
"color": "orange",
|
||||
"index": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(info|INFO|Info)$",
|
||||
"result": {
|
||||
"color": "green",
|
||||
"index": 3
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "regex",
|
||||
"options": {
|
||||
"pattern": "^(debug|DEBUG|Debug)$",
|
||||
"result": {
|
||||
"color": "blue",
|
||||
"index": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": {
|
||||
"type": "color-text"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Caller"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 200
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Component id"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 200
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "Raw record"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 260
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
}
|
||||
"showHeader": true,
|
||||
"cellHeight": "sm",
|
||||
"footer": {
|
||||
"show": false,
|
||||
"reducer": [
|
||||
"sum"
|
||||
],
|
||||
"countRows": false,
|
||||
"fields": ""
|
||||
},
|
||||
"sortBy": [
|
||||
{
|
||||
"displayName": "Time",
|
||||
"desc": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "Prometheus, Loki, Tempo, Grafana and Alloy all write logfmt, so one selective parser covers the whole stack. Selective on purpose: a bare `| logfmt` would explode Loki's own query-metrics lines into dozens of columns."
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "row",
|
||||
"title": "Raw log lines",
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 10,
|
||||
"type": "logs",
|
||||
"title": "Raw log lines",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{namespace=\"observability\"} |~ \"(?i)(error|level=error|fatal)\"",
|
||||
"queryType": "range"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Descending",
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true
|
||||
},
|
||||
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -64,19 +64,42 @@ datasources:
|
||||
datasourceUid: prometheus
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
# Span -> logs by trace id rather than by service name.
|
||||
#
|
||||
# The service-name mapping this replaces resolved `service.name` to Loki's
|
||||
# `app` label, which Alloy derives from app.kubernetes.io/name. That label
|
||||
# exists for keycloak and classic-models-api but not for the six Kong
|
||||
# gateways or librechat — and the gateways are precisely where the trace
|
||||
# id does appear in the logs. The mapping pointed at the services where it
|
||||
# could not work.
|
||||
#
|
||||
# A custom query on ${__trace.traceId} sidesteps the label entirely: every
|
||||
# source that writes the id anywhere in the line is matched. The stream
|
||||
# selector is spelled out rather than `.+` so the query stays bounded.
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: -5m
|
||||
spanEndTimeShift: 5m
|
||||
filterByTraceID: true
|
||||
spanStartTimeShift: -1h
|
||||
spanEndTimeShift: 1h
|
||||
filterByTraceID: false
|
||||
filterBySpanID: false
|
||||
tags:
|
||||
- key: service.name
|
||||
value: app
|
||||
customQuery: true
|
||||
query: '{job=~"kong-audit|keycloak/keycloak|classic-models/classic-models-api|librechat/librechat"} |= "$${__trace.traceId}"'
|
||||
# Span -> metrics against the span metrics Tempo's own metrics_generator
|
||||
# feeds into Prometheus. Without explicit queries this block renders a
|
||||
# bare datasource link and nothing else, which is what it did before.
|
||||
# $$__tags expands to the tag mapping below, so each query is scoped to
|
||||
# the span's own service.
|
||||
tracesToMetrics:
|
||||
datasourceUid: prometheus
|
||||
spanStartTimeShift: -5m
|
||||
spanEndTimeShift: 5m
|
||||
spanStartTimeShift: -30m
|
||||
spanEndTimeShift: 30m
|
||||
tags:
|
||||
- key: service.name
|
||||
value: service
|
||||
queries:
|
||||
- name: 'Request rate'
|
||||
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags}[$$__rate_interval]))'
|
||||
- name: 'Error rate'
|
||||
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags,status_code="STATUS_CODE_ERROR"}[$$__rate_interval]))'
|
||||
- name: 'Latency p95'
|
||||
query: 'histogram_quantile(0.95, sum by (span_name, le) (rate(traces_spanmetrics_latency_bucket{$$__tags}[$$__rate_interval])))'
|
||||
|
||||
Reference in New Issue
Block a user