stt/infra-observability
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: stt/infra-observability:940abdf6fcaee91215089a52eb2a782d06e7f1a8
Branches Tags
stt/infra-observability:main
..
compare: stt/infra-observability:main
Branches Tags
stt/infra-observability:main

18 Commits
940abdf6fc .. main

Author SHA1 Message Date
Stéphane Tailland ad419851c6 Expose only the tools this stack has datasources for 
The server ships 65 tools by default. An MCP client sends the whole tool
list on every turn, so that was 75 KiB — measured at ~19k prompt tokens
per request before the user has asked anything. A three-turn conversation
therefore tripped Kong's ai-rate-limiting-advanced, 50k tokens per minute
per consumer, and the assistant sat in a 429 retry loop: 15 rejected calls
for one that got through. Proven to be Kong and not the provider —
upstream_status empty, tries empty, latencies.proxy -1, and x-ai-ratelimit
headers on the response.

Down to 17 tools and ~5.4k tokens, a 72% cut, keeping query_prometheus,
query_loki_logs, the label listings and dashboard search.

Dropped: everything Cloud-only on OSS Grafana (incident, oncall, sift,
asserts, pyroscope), every datasource this cluster does not run (athena,
clickhouse, cloudwatch, influxdb, elasticsearch, graphite, quickwit,
snowflake), and the administration surface (provisioning, plugin, api,
config, folder, annotations, snapshot, rendering, navigation).
--disable-write on top: the service account is Viewer, so writes would
fail anyway, and this keeps their definitions out of every request.

Note what this does not fix: list_prometheus_metric_names still returns
all 3330 metric names, 137 KiB, and the assistant called it twice while
exploring. The rate limit did its job by refusing those calls before they
reached the provider — raising it would have paid for the same behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 17:45:34 +02:00
Stéphane Tailland 6d020572e0 Take the observability secrets from Vault 
k8s/secret.yaml held the Grafana admin credentials and, since the MCP
server landed, its service account token — gitignored, so deploying this
stack required the right file to exist on the right laptop. It now comes
from Vault through External Secrets, and nothing sensitive has to exist
locally.

A dedicated store rather than reusing vault-kong-v2: that one's AppRole
carries kong-v2-readonly, scoped to secrets/data/kong-v2/*, so it cannot
see this path — confirmed with a probe ExternalSecret that failed with
"could not get secret data from provider". Widening that policy would have
handed the Kong reader observability's secrets for no reason. So
observability-readonly and observability-eso were created in Vault,
mirroring the existing pattern exactly.

The AppRole credentials stay out of git; the ClusterSecretStore comments
carry the commands that mint them.

dataFrom.extract pulls the whole path instead of naming each key, so a new
key in Vault reaches the cluster with no edit here. The target keeps the
name every consumer already uses, so neither the Grafana Deployment nor
mcp-grafana changed — verified, neither pod restarted and the token in the
cluster matches Vault byte for byte.

secret.example.yaml no longer describes a file to create; it documents
where the keys live and how the Grafana token is minted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 17:27:15 +02:00
Stéphane Tailland 9bf9ecc358 Add Grafana's MCP server 
Exposes Prometheus, Loki and Tempo as tools an LLM can call, so a question
like "what failed in the last transaction" is answered by querying rather
than by building another panel — which is where the trace dashboard kept
running out of road.

Works against OSS Grafana; the Cloud-only tools (Incidents, OnCall, Sift,
Agent Observability, Assistant) are simply absent. Verified end to end:
the MCP handshake succeeds and list_datasources returns Loki, Prometheus
and Tempo.

Authenticates with a Grafana service account token, Viewer role, created
once through /api/serviceaccounts and stored in the gitignored
k8s/secret.yaml. secret.example.yaml documents how to mint it, since
Grafana returns the value only at creation.

Two things the deployment needs that are not obvious:

  * runAsUser must be numeric. The image declares USER mcp-grafana by
    name, Kubernetes cannot resolve that to check it is not root, and
    refuses to start the container under runAsNonRoot with "image has
    non-numeric user". 1000/999 is what the image actually uses.

  * --allowed-hosts must name the Service. The server validates the Host
    header against a loopback-only default as DNS-rebinding protection, so
    the first in-cluster call returned "forbidden: host not allowed".
    Named rather than disabled with "*": the check is worth keeping.

Deliberately in-cluster only, with no OIDC and no audit trail — anything
that can reach this Service reads all the telemetry. The intended end
state is the same treatment as the five domain MCP servers, behind a Kong
gateway with openid-connect, consumer groups and http-log.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 17:21:32 +02:00
Stéphane Tailland e6a3c53c8b Drop the span filter that could not work 
I set spanNameOperator "!~" with a pattern and said it was verified. It
was not: I had checked that the option names exist in Grafana's bundle,
not that the behaviour was what I claimed, and then wrote that it was no
longer a supposition. It shipped a filter chip that hid nothing — the
panel reported "212 matches" with every middleware span still listed.

Reading the implementation: spanName is compared with an exact string
equality against operationName, so only "=" and "!=" against a single
literal name do anything. It is also pushed into the tag-filter list,
where "=~"/"!~" are honoured — but that matcher looks the key up among
the span's tags, and a span's name is not a tag, so the pattern matches
nothing and the negative filter passes everything.

So pattern-based exclusion is not available in this panel. Remove the
option rather than leave a chip that suggests filtering is happening, and
say so in the description, pointing at the built-in Critical path button
as the one lever that does narrow the view.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 13:45:19 +02:00
Stéphane Tailland 32995ff949 Hide framework plumbing in the trace view by default 
A chat transaction is 212 spans, and almost all of them are Express
middleware, mongoose and mongodb calls, kong.dns lookups and raw SQL.
The flow — LibreChat to the gateway to the backend — is buried in it.

Set a default span filter on the panel: spanNameOperator "!~" against
those names with matchesOnly, so the view opens on the spans that
describe the exchange. "Show all spans" and clearing the filter both
bring them back.

Deliberately a display filter, not a source change. The spans are still
produced and still stored; only this panel's default view is quieter.
It is also the only lever that covers the LibreChat ones: LibreChat does
not use auto-instrumentations-node — it registers express, mongodb and
mongoose individually in its own code — so no environment variable
disables them, and they are the bulk of the 212.

The option names were checked against Grafana's own traces panel bundle
rather than assumed: spanName, spanNameOperator, serviceName,
serviceNameOperator, matchesOnly and criticalPathOnly all exist there,
and "!~" is among the operators it offers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 13:42:10 +02:00
Stéphane Tailland fd4ac86d4a Select transactions by what starts them, not by span count 
The count-based filter let LibreChat's own /metrics scrapes through: 50
of them in six hours, crowding out the two real conversations. Combining
the count with a root-service match does not work either — TraceQL drops
the aggregate when it is joined to a spanset conjunction, verified: the
scrapes came back despite having a single server span.

Root on the service that starts a transaction instead, and require it to
reach a second one:

  {trace:rootService="librechat"} && {resource.service.name!="librechat"}

Eight traces over six hours against fifty, and all eight are real: the
chat requests, the MCP OAuth callbacks, the OIDC login callback. Simpler
than counting spans and stricter.

The trade is explicit in the panel description: anything not entering
through LibreChat is invisible — a direct call to a gateway, a curl
against an API. "Everything" is still in the dropdown for those.

Worth revisiting when the agents replace LibreChat as the entry point,
since the root service is then no longer librechat.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 13:32:37 +02:00
Stéphane Tailland 5be2c76d94 Join identity events to a trace through the user id 
Kong does not propagate the trace when its openid-connect plugin calls
Keycloak: verified by tracing a request end to end and watching the
TOKEN_EXCHANGE it triggered land under Keycloak's own trace id instead.
It is not a missing setting — Kong's propagation module injects through
kong.service.request.*, which is the proxied upstream request, while the
plugin reaches Keycloak with its own HTTP client. Nothing in the
opentelemetry schema or kong_defaults.lua covers that path.

There is a join key even so: the audit record's
authenticated_entity.id is the same value as Keycloak's userId. So add a
panel listing the identities Kong authenticated on the selected trace —
usually more than one, LibreChat's own on the llm hop and the end user's
on the domain hops — and let a click on one scope the Keycloak panel to
that user.

A Grafana query variable cannot derive this automatically: Loki's
label/values endpoint rejects a pipeline ("only label matchers are
supported"), so the extracted field cannot feed a variable. Hence the
click.

That gives the intended chain for one user: LOGIN, CODE_TO_TOKEN,
USER_INFO_REQUEST, then a TOKEN_EXCHANGE per audience — verified against
live data.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 12:55:51 +02:00
Stéphane Tailland e8b600d203 Rebuild the trace dashboard around the trace itself 
Three things were wrong with it, two of them mine.

Clicking a trace id emptied the trace list. The data link passed
var-traceFilter=${traceFilter:queryparam}, but that formatter already
emits "var-name=value" — prefixing it again wrote the literal string
"var-traceFilter={kind=server} | count() > 2" into the variable, which is
not valid TraceQL, so the panel errored. The link now sets only
var-traceId.

The spans were split across two tables, HTTP hops and Kong plugins. They
are one tree, and splitting them made the order unreadable. Replaced by
Grafana's own trace view: the full waterfall, every span nested in order
across LibreChat, Keycloak, the gateways and the backend, with span
attributes on click and the built-in jump to that span's logs.

And the dashboard never showed the identity side, which is most of the
point of following a request through this stack. Keycloak's event log
carries it: the token exchanges with the audience minted, the requesting
client, the end user, the grant type and the scopes. Parsed into columns
with one independent regexp per field — chaining json into line_format
into another parser has already cost a 60s timeout here, and each event
type carries a different subset of the keys, so an all-or-nothing parse
silently drops rows. The escaping is not incidental: the line is JSON, so
the message reads type=\"...\" and the pattern has to match the
backslash.

That panel is scoped to the time range rather than to the selected trace,
on purpose. Kong's openid-connect plugin calls Keycloak without
propagating the trace, so its token exchanges carry their own trace id
and scoping them to the transaction would show nothing. LibreChat's own
calls to Keycloak do propagate and do appear under the transaction —
verified both ways.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 12:30:20 +02:00
Stéphane Tailland 3988df4916 Select traces by component count, not by service 
The first version filtered the trace list by service name, which does not
remove the noise it was built to remove: Keycloak token calls and gateway
401s are services too, so with Service=All the panel showed exactly the
health-check traffic that makes Explore unusable, and the real LibreChat
transaction was nowhere in the first 50 rows.

Count SERVER spans instead. A token call has one, a health probe has one,
a gateway 401 has two; a real end-to-end transaction has four. So
{kind=server} | count() > 2 isolates them, and does it without naming a
single service — nothing to edit when an agent or a domain is added.
Verified against live data: it returns the LibreChat chat transactions
and nothing else.

The span panels were wrong in the same way: they showed the spans of
every trace at once, so the two tables were filled with unrelated
gateways. They now scope to {trace:id="$traceId"}, set by clicking a row
above, and sit empty until one is picked — which is the honest state, not
a wall of noise.

The service variable is gone; the trace filter is a named dropdown
instead, with the count-based default and a couple of looser options.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 07:53:47 +02:00
Stéphane Tailland c95af8209e Add a Distributed Traces dashboard 
Explore's unfiltered trace search is unusable on this cluster: Tempo does
not return {} newest-first, so raising the limit does not walk backwards
in time and the health-check traffic — 91 Keycloak token calls and dozens
of /health/ and /metrics probes an hour — crowds real transactions out of
the first 20 results. Measured: at a fixed instant, limit=20 returned
06:50:43-06:51:07 while limit=100 returned 06:47:44-06:50:02.

Four panels:
  * the traces on the dashboard's time range, filtered by service, each
    row linking its trace id into the traceId variable;
  * the HTTP hops between components — the SERVER-kind spans, one per
    service entry, with method/status/route projected via select(), so
    the Express middleware, DNS and SQL spans stay out;
  * the Kong plugin spans with their individual durations;
  * the Loki lines carrying the selected trace id.

No metric panels, on purpose. A span's own duration is already in the
span tables and is measured rather than estimated; aggregate RED by span
name is what obs-classic-models-api, obs-kong-gateways and obs-librechat
already show. Neither answers "the metrics of this span", so putting them
here only duplicated existing dashboards.

Two correlation limits are written into the panel descriptions rather
than left to be discovered: the join is by trace id and not span id
because no emitter logs a span id, and LibreChat logs carry no trace id
at all so its lines never appear.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-27 07:44:09 +02:00
Stéphane Tailland d3abde1040 Drop the traces dashboard, keep the datasource wiring 
The dashboard reimplemented, less well, what Explore already does: its search
panel had no TraceQL query builder and no autocompletion, and its service
dependency table replaced an interactive node graph with a list. Searching and
exploring traces belongs in Explore.

The datasource correlations this work also fixed are kept — they are what was
actually missing, and they apply in Explore exactly as they would in a
dashboard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-26 07:55:17 +02:00
Stéphane Tailland 367ebd1da5 Add a traces dashboard and make the span correlations work 
Trace search lived only in Explore. `obs-traces` brings it into a dashboard:
TraceQL assembled from a service selector, a minimum duration, a status and a
free-text field, plus the aggregate RED view from the span metrics Tempo derives
from the traces themselves.

The status and free-text variables hold TraceQL fragments rather than values, so
an empty one contributes nothing and the query stays valid — the same reason the
log dashboards inject their search term the way they do.

Both span correlations were declared but non-functional.

tracesToLogsV2 mapped service.name onto Loki's `app` label, which Alloy derives
from app.kubernetes.io/name. That label exists for Keycloak and the Classic
Models API but not for the six Kong gateways or LibreChat — and the gateways are
the only place the trace id actually appears in the logs, so the mapping pointed
squarely at the services where it could not resolve. Replaced with a custom
query on ${__trace.traceId}, which matches any source writing the id anywhere in
the line.

tracesToMetrics had no queries at all and rendered a bare datasource link. It
now carries rate, error-rate and p95 against traces_spanmetrics_*, scoped to the
span's own service.

The service graph stays a table: serviceMap is a query type Grafana's frontend
assembles for Explore and returns no frames through the dashboard query API.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-26 07:44:57 +02:00
Stéphane Tailland a2f5d725a3 Make the Keycloak identity events table actually render 
The panel never displayed: its query combined `line_format` with `logfmt`, and
on this data that pairing is pathological. Measured on a 24h range, the stream
selector with its json filter answers in 0.7s, adding line_format alone 0.3s,
adding logfmt alone 0.2s — chained, 60s and a timeout. The bar panels above it
survived because an aggregation streams; the log query did not.

Replaced with one independent `regexp` per field, reading the key="value" pairs
straight out of the JSON message without reformatting the line. 0.1s, and the
raw record now stays raw since nothing rewrites it.

Independent stages also fix a correctness problem the single-regexp version
would have had: the field set differs per event type — LOGIN_ERROR carries
`error` and `grant_type`, USER_SESSION_DELETED carries neither — so one regexp
spanning them all would drop whole rows on a missing field. All four types
observed on this cluster now extract correctly.

Failed and successful logins widen from LOGIN_ERROR and LOGIN to the *_ERROR
and CODE_TO_TOKEN families: a failed refresh or code exchange is the same
signal, and TOKEN_EXCHANGE dominates this cluster's traffic.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 19:05:50 +02:00
Stéphane Tailland 7c416ce06f Fix column order and byte counts in the log tables 
Three defects, all mine.

Column order was ignored because organize's indexByName was keyed on the
renamed display labels. Grafana keys it on the original field names, so the
mapping matched nothing and the order fell back to whatever order the frames
arrived in — which is why the raw line landed second. Every table is affected,
not just the access one.

The byte count was wrong for the NCSA sources: a trailing `pattern` capture
takes the rest of the line, so `<size>` held `16 "-" "kube-probe/1.35"` rather
than 16, and for nginx it swallowed the whole referer and user-agent. Replaced
with a `regexp` parser bounding the capture on \d+, which also handles both
line shapes — Keycloak's access line ends at the size while gunicorn's and
nginx's carry more fields after it.

Access log columns are now time, status, method, path, source, client, latency,
bytes, raw line, with the raw record last in every table.

gunicorn and nginx now share one query since their format is identical, which
also halves the duplicate class names the status panel showed — one series per
query returning the same class.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 18:56:27 +02:00
Stéphane Tailland eb35419ad0 Split log search into access, application and audit dashboards 
One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.

Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.

The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.

Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.

Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 18:48:11 +02:00
Stéphane Tailland 40ffac4acf Add a consolidated log search dashboard 
The per-application dashboards answer "how is this component behaving". This
one answers "where did that string come from", across every source at once, and
is kept separate so neither has to compromise for the other.

Scope is a curated preset rather than a free namespace picker: the cluster
carries 48 log sources and most are infrastructure noise. The presets are job
regexes over the <namespace>/<container> label Alloy sets — except kong-audit,
which Fluent Bit ships with no namespace at all. That mismatch between the two
collection paths is the reason a single naive selector cannot span both, and
why the presets spell the sources out.

The search term is injected inside LogQL backticks, a raw string, so a quote
typed in the box cannot break the query; an empty box yields `(?i)`, a valid
regex matching every line, which makes "no search" the default rather than a
special case. The four stat panels carry `or vector(0)` so an empty result
reads as 0 rather than "No data" — on a search box, finding nothing is an
answer, not a failure.

Two deliberate omissions. No per-format parsing: the Message column holds the
line as emitted, because the five formats agree on nothing and that is what the
per-application dashboards are for. And the level filter is partial by nature —
roughly 40% of lines are `unknown`, since access lines and Kong records carry
no severity for Loki to infer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 18:23:48 +02:00
Stéphane Tailland 7a8a476ef0 Show parsed log tables instead of raw lines 
Raw lines are close to unreadable when the payload is JSON, and the Kong audit
records are the worst case: a wall of braces where the interesting fields are
status, route and latency.

Each dashboard now leads with a table carrying one column per parsed field,
sortable and filterable, with the unparsed record as the last column and the
cell inspector enabled on every cell. The original logs panel moves into a
collapsed row underneath rather than disappearing: a table cannot render ANSI
colour, expand a record inline, tail live, or turn a trace id into a link to
Tempo.

Parsing is per-format since the five sources agree on nothing — nested JSON for
Kong, JSON for Keycloak, NCSA access lines for the Django API, ANSI-coloured
text for LibreChat, logfmt for the stack itself.

The Kong and stack parsers name their fields explicitly rather than extracting
everything: a bare `| json` on a Kong record yields around seventy columns, and
a bare `| logfmt` explodes Loki's own query-metrics lines into dozens. Columns
are selected by include-pattern rather than by excluding names, because the
stream labels that ride along are not a fixed set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 18:11:53 +02:00
Stéphane Tailland 0ba38bc8f7 Add a Kong gateways dashboard 
One dashboard for all six gateways, with a selector accepting one, several or
all of them.

The selector has to bridge a mismatch: metrics carry the full gateway name
(catalog-dev) while audit logs carry the short domain (catalog), because the
former comes from the scrape job's label and the latter from the URI path each
gateway posts its http-log to. The variable holds the short form, PromQL
re-adds the -dev suffix and LogQL uses it as-is, so a single control drives
both signals. The capture group in the variable's regex also excludes the
unrelated `kong` gateway, which has no -dev suffix.

Latency is split three ways rather than shown as one number: total request
latency, time spent inside Kong, and upstream latency. The gap between the
first and the last is what the gateway itself costs, which is the question
worth asking of a proxy.

Status classes come from label_replace on the `code` label for metrics and
from LogQL's json parser for the audit records, since those carry no severity
of their own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-25 18:02:07 +02:00
21 changed files with 6625 additions and 93 deletions
Expand all files Collapse all files
Makefile
+9 -4
View File
@@ -46,7 +46,8 @@ help: ## Show this help
.PHONY: deploy
deploy: check-context secret ## Deploy the whole stack (runtimes + dashboards)
$(KUBECTL) apply -k $(K8S)
# --server-side for the same reason as the dashboards target below.
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts
$(MAKE) --no-print-directory wait
.PHONY: deploy-runtime
@@ -89,7 +90,11 @@ secret: ## Generate k8s/secret.yaml (random admin password) if absent
.PHONY: dashboards
dashboards: check-context dashboards-check ## Apply dashboards only (no Grafana restart)
$(KUBECTL) apply -k $(DASHBOARDS)
# Server-side apply: client-side apply stores a full copy of the object in the
# kubectl.kubernetes.io/last-applied-configuration annotation, which is capped at
# 256KB. The dashboards passed that in aggregate. Server-side apply tracks field
# ownership in metadata instead, so the ceiling becomes etcd's ~1MB per object.
$(KUBECTL) apply -k $(DASHBOARDS) --server-side --force-conflicts
printf "$(C_OK)grafana-dashboards ConfigMap updated$(C_OFF) -- visible in Grafana within ~70s\n"
printf " (kubelet volume sync ~60s + provisioner poll 10s, no restart).\n"
printf " For immediate effect: $(C_WARN)make restart-grafana$(C_OFF)\n"
@@ -310,7 +315,7 @@ _flush:
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=0
$(KUBECTL) -n $(NS) wait --for=delete pod -l app=$(COMP) --timeout=120s 2>/dev/null || true
$(KUBECTL) -n $(NS) delete pvc $(COMP)-data --ignore-not-found --wait=true
$(KUBECTL) apply -k $(K8S)/$(COMP)
$(KUBECTL) apply -k $(K8S)/$(COMP) --server-side --force-conflicts
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=1
##@ Validation and teardown
@@ -326,7 +331,7 @@ diff: check-context ## Diff the manifests against the cluster state
.PHONY: validate
validate: dashboards-check ## Validate the manifests (build + server dry-run)
kustomize build $(K8S) >/dev/null && printf "$(C_OK)kustomize build OK$(C_OFF)\n"
$(KUBECTL) apply -k $(K8S) --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
.PHONY: undeploy
undeploy: ## DESTRUCTIVE: delete the stack, PVCs kept (CONFIRM=yes)
README.md
+144
View File
@@ -132,6 +132,13 @@ forces a pod restart when they change.
| `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
| `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |
> **Applies are server-side.** Client-side `kubectl apply` stores a full copy of
> each object in the `kubectl.kubernetes.io/last-applied-configuration`
> annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate
> at eight dashboards. `--server-side` tracks field ownership in metadata
> instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one
> ConfigMap per dashboard is the next step if that is ever reached.
StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
use `strategy: Recreate`: with an RWO volume the old pod must release it before
the new one can start.
@@ -191,6 +198,14 @@ changes in `container_start_time_seconds`.
| `librechat` | OTel SDK → `tempo:4318` | built-in `/metrics`, Bearer token | stdout | `obs-librechat` |
| `classic-models-api` | `opentelemetry-instrument` → `tempo:4318` | django-prometheus `/metrics` | JSON + `otelTraceID` | `obs-classic-models-api` |
The six Kong gateways share a single dashboard, `obs-kong-gateways`, with a
`gateway` selector at the top (one, several, or all). Its variable resolves a
subtlety of this setup: the metrics label is the full gateway name
(`catalog-dev`) while the audit-log label is the short domain (`catalog`). The
variable holds the short form, PromQL re-adds the `-dev` suffix, and LogQL uses
it as-is — so one control drives both signals. The capture group in its regex
also drops the unrelated `kong` gateway, which carries no `-dev` suffix.
Their manifests live in the `infra-keycloak`, `infra-librechat` and
`classic-models-api` repositories; this repository only holds the platform.
@@ -198,6 +213,135 @@ Each application dashboard carries the same four sections: an overview of RED
indicators, HTTP traffic, a runtime section specific to the stack in use, and a
**Logs** row, followed by container resources.
### Distributed tracing
Traces are searched and explored in **Explore**, on the Tempo datasource, not
from a dashboard. Explore's TraceQL query builder, autocompletion, trace view
and interactive service graph have no dashboard equivalent worth maintaining —
a dashboard search panel is a worse copy of a tool that already exists.
What the repository does own is the datasource wiring, which applies in Explore
exactly as it would in a dashboard.
#### Span-to-logs correlation
The Tempo datasource resolves a span's logs with a custom query on
`${__trace.traceId}` rather than by mapping `service.name` to a Loki label. The
label mapping it replaces resolved to `app`, which Alloy derives from
`app.kubernetes.io/name`: that exists for Keycloak and the Classic Models API
but not for the six Kong gateways — and the gateways were the only place the
trace id actually appeared. The mapping pointed at the services where it could
not work.
Coverage today, each verified by taking a trace id and searching for it:
| Service | Trace id in logs | How |
|---|---|---|
| Kong gateways (×6) | yes | `x-trace-id` in the http-log record |
| Keycloak | yes | MDC, via `KC_LOG_MDC_ENABLED` |
| classic-models-api | yes | inbound `traceparent` in the gunicorn access line |
| LibreChat | no | would need the upstream image patched |
#### Span-to-metrics correlation
`tracesToMetrics` had no queries defined and rendered a bare datasource link. It
now carries rate, error-rate and p95 queries against `traces_spanmetrics_*`,
scoped to the span's own service.
Those span metrics are worth knowing about independently: Tempo's
`metrics_generator` derives them from the traces and remote-writes them to
Prometheus, keyed by `service` and `span_name`. They measure the *span*, so a
slow database statement appears next to a slow HTTP handler — a granularity the
gateway and per-application dashboards, which measure HTTP, do not have.
### Log dashboards, split by kind
Logs are searched from three dashboards rather than one, because the three
questions have nothing in common and a single view compromised on all of them.
| Dashboard | Question | Sources |
|---|---|---|
| `obs-access-logs` | Which HTTP requests came in? | Kong records, Keycloak access logger, Django gunicorn, ingress-nginx |
| `obs-application-logs` | What is the code reporting, and what broke? | Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log |
| `obs-audit-logs` | Who authenticated, what changed, who called what? | Keycloak `org.keycloak.events`, Kong http-log records |
Each source keeps its own parser and they all emit the same field names, so the
frames merge into one table with consistent columns.
Keycloak's identity events are read with one independent `regexp` per field
rather than `line_format` piped into `logfmt`. Two reasons, both measured: that
combination is pathological on this data — each stage answers in 0.3s alone,
chained they time out at 60s and the table never renders — and the field set
differs per event type, so a single regexp spanning them would fail whole rows
on a missing field. Independent stages leave the absent field empty and keep the
row, and the query returns in 0.1s. Access logs converge on
method / path / status / client; application logs on level / message.
Three things this split fixes that a single consolidated view could not:
**The kinds are subtracted, not just filtered.** Application logs exclude
Keycloak's `access-log` and `events` loggers by name, the Django container's
NCSA lines by shape, and the Kong containers' stray access lines by requiring
the nginx date prefix. Without that, access lines drown the signal.
**Levels are read from the line, not inferred.** Loki's `detected_level` marks
78% of the Kong gateway output `unknown` even though every one of those lines
states `[notice]` or `[warn]`. Each parser extracts the level and lowercases it,
so nginx's `notice`, Keycloak's `INFO` and LibreChat's `info` land on one scale
— and the level selector has no `unknown` entry, because a line reaching the
table always states one.
**The scope selector actually scopes.** Each query carries both
`job=~"${source:regex}"` and a literal job matcher. With a single selector,
choosing "All" widened the query back to every stream in the cluster — which is
how MongoDB and cert-manager ended up in an application view.
Latency is populated for Kong only: the NCSA format the other sources use
carries no timing field. That is a property of the format, not a gap in the
query.
The NCSA sources are read with `regexp`, not `pattern`. A trailing `pattern`
capture consumes the rest of the line, so a `<size>` at the end swallowed the
referer and user-agent along with the byte count. Bounding the capture on `\d+`
stops it at the number, and `regexp` tolerates both shapes — Keycloak's access
line ends at the size, gunicorn's and nginx's carry more fields after it.
In every table the raw record is the last column, and the `organize`
transformation's `indexByName` keys on the original field names rather than the
renamed labels — keying it on the display names silently leaves the column order
to whatever order the frames arrived in.
### Parsed log tables### Parsed log tables
Every dashboard shows its logs twice: a **table** with one column per parsed
field, and, in a collapsed row underneath, the original **logs panel**.
The table is what makes a JSON record readable — status, method, route, latency
as sortable, filterable columns rather than a wall of braces. Its last column
holds the unparsed record, and every cell has the inspector enabled, so the raw
line is one click away.
The logs panel stays because a table cannot do what it does: render ANSI colour,
expand a record inline, tail live, and turn a `trace_id` into a link to Tempo.
It is collapsed so it costs nothing until needed.
Parsing is per-format, because these five sources agree on nothing:
| Source | Format | Parser |
|---|---|---|
| Kong audit | nested JSON | `\| json` with explicit field paths |
| Keycloak | JSON | `\| json` |
| classic-models-api | NCSA access lines | `\| pattern` |
| LibreChat | ANSI-coloured text | `line_format` to strip ANSI, then `\| pattern` |
| Observability stack | logfmt | `\| logfmt` with named fields |
Two choices worth knowing. The Kong and stack parsers name their fields
explicitly rather than extracting everything: a bare `\| json` on a Kong record
yields about seventy columns, and a bare `\| logfmt` explodes Loki's own
query-metrics lines. And `\| pattern` keeps lines it cannot match, with empty
columns, rather than dropping them — so the Django table never silently hides
an application log among the access lines.
The Logs row leans on Loki's `detected_level`, which is inferred at ingestion
time, so a single query works across all three log formats — Keycloak emits JSON,
LibreChat colour-coded text, and the Django API plain gunicorn access lines. No
k8s/grafana/dashboards/kustomization.yaml
+5
View File
@@ -17,8 +17,13 @@ configMapGenerator:
- name: grafana-dashboards
files:
# >>> dashboards (generated by `make dashboards-sync`) >>>
- obs-access-logs.json
- obs-application-logs.json
- obs-audit-logs.json
- obs-classic-models-api.json
- obs-distributed-traces.json
- obs-keycloak.json
- obs-kong-gateways.json
- obs-librechat.json
- obs-stack-health.json
# <<< dashboards <<<
k8s/grafana/dashboards/obs-access-logs.json
+846
View File
@@ -0,0 +1,846 @@
{
"uid": "obs-access-logs",
"title": "Access Logs",
"description": "One row per HTTP request, across the Kong gateways, Keycloak, the Classic Models API and the ingress controller. Each source is parsed by its own format and normalised to the same columns.",
"tags": [
"observability",
"logs",
"access"
],
"editable": false,
"schemaVersion": 39,
"version": 1,
"refresh": "",
"time": {
"from": "now-1h",
"to": "now"
},
"timezone": "browser",
"templating": {
"list": [
{
"name": "source",
"label": "Source",
"type": "custom",
"query": "Kong gateways : kong-audit,Keycloak : keycloak/keycloak,Classic Models API : classic-models/classic-models-api,Ingress NGINX : ingress-nginx/controller",
"multi": true,
"includeAll": true,
"allValue": "kong-audit|keycloak/keycloak|classic-models/classic-models-api|ingress-nginx/controller",
"current": {
"text": [
"All"
],
"value": [
"$__all"
]
}
},
{
"name": "search",
"label": "Search",
"type": "textbox",
"query": "",
"current": {
"text": "",
"value": ""
},
"description": "Case-insensitive regex over the whole line. Empty matches everything."
}
]
},
"panels": [
{
"id": 1,
"type": "stat",
"title": "Requests",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum(count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range])) or vector(0)",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "blue"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"description": "Every line the selected sources emitted over the dashboard's time range."
},
{
"id": 2,
"type": "stat",
"title": "Client errors (4xx)",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | status =~ \"4..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"4..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"4..\" [$__range])) or vector(0))",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
},
{
"color": "orange",
"value": 1
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"description": "Summed across all four sources, each parsed by its own format."
},
{
"id": 3,
"type": "stat",
"title": "Server errors (5xx)",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | status =~ \"5..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"5..\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | status =~ \"5..\" [$__range])) or vector(0))",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 4,
"type": "stat",
"title": "Sources active",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "count(sum by (job) (count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range]))) or vector(0)",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "purple"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "purple",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 5,
"type": "timeseries",
"title": "Requests by source",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 4
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"mappings": [],
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "bars",
"lineWidth": 1,
"fillOpacity": 80,
"showPoints": "never",
"barAlignment": 0,
"stacking": {
"mode": "normal",
"group": "A"
}
}
},
"overrides": []
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"sum"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "Bars rather than lines: a request count per bucket is a run of discrete events."
},
{
"id": 6,
"type": "timeseries",
"title": "Status classes",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\" | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
"legendFormat": "{{class}}",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
"legendFormat": "{{class}}",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (class) (count_over_time({job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)` | label_format class=`{{ printf \"%.1sxx\" .status }}` [$__auto]))",
"legendFormat": "{{class}}",
"queryType": "range"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"mappings": [],
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "bars",
"lineWidth": 1,
"fillOpacity": 80,
"showPoints": "never",
"barAlignment": 0,
"stacking": {
"mode": "normal",
"group": "A"
}
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "2xx"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "green"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "3xx"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "blue"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "4xx"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "orange"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "5xx"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "red"
}
}
]
}
]
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"sum"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "Derived from each source's parsed status, so the formats land on one scale. A class can appear once per query in the legend; the stack itself is the correct total."
},
{
"id": 7,
"type": "table",
"title": "Requests",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 14,
"w": 24,
"x": 0,
"y": 12
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\"",
"queryType": "range",
"maxLines": 1000
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
"queryType": "range",
"maxLines": 1000
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
"queryType": "range",
"maxLines": 1000
}
],
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "merge",
"options": {}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|status|method|path|job|client|latency|size)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"status": 1,
"method": 2,
"path": 3,
"job": 4,
"client": 5,
"latency": 6,
"size": 7,
"body": 8
},
"renameByName": {
"timestamp": "Time",
"body": "Raw line",
"status": "Status",
"method": "Method",
"path": "Path",
"job": "Source",
"client": "Client",
"latency": "Latency (ms)",
"size": "Bytes"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Status"
},
"properties": [
{
"id": "custom.width",
"value": 80
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^2..$",
"result": {
"color": "green",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^3..$",
"result": {
"color": "blue",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^4..$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^5..$",
"result": {
"color": "red",
"index": 3
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Method"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "Source"
},
"properties": [
{
"id": "custom.width",
"value": 230
}
]
},
{
"matcher": {
"id": "byName",
"options": "Client"
},
"properties": [
{
"id": "custom.width",
"value": 130
}
]
},
{
"matcher": {
"id": "byName",
"options": "Latency (ms)"
},
"properties": [
{
"id": "custom.width",
"value": 110
}
]
},
{
"matcher": {
"id": "byName",
"options": "Bytes"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw line"
},
"properties": [
{
"id": "custom.width",
"value": 240
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "One row per request across every source. Latency is populated for Kong only — the NCSA format the others use carries no timing field, which is a property of the format rather than a gap in the query. Raw line holds the record as emitted."
},
{
"id": 8,
"type": "logs",
"title": "Requests as a log stream",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 26
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"kong-audit\"} |~ `(?i)$search` | json method=\"request.method\", path=\"request.uri\", status=\"response.status\", client=\"client_ip\", latency=\"latencies.request\", size=\"response.size\"",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"keycloak/keycloak\"} |~ `(?i)$search` | json logger=\"loggerName\", m=\"message\" | logger=\"org.keycloak.http.access-log\" | line_format `{{.m}}` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"classic-models/classic-models-api|ingress-nginx/controller\"} |~ `(?i)$search` | regexp `^(?P<client>\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"(?P<method>\\S+) (?P<path>\\S+) [^\"]*\" (?P<status>\\d{3}) (?P<size>\\d+)`",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": true,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Same result set. Kept for live tail and for the trace link the Loki datasource derives from a trace id, neither of which a table can do."
}
]
}
k8s/grafana/dashboards/obs-application-logs.json
+926
View File
@@ -0,0 +1,926 @@
{
"uid": "obs-application-logs",
"title": "Application Logs",
"description": "What the applications themselves report, with access lines and audit events subtracted. Levels are read from the line rather than inferred. Defaults to errors and warnings.",
"tags": [
"observability",
"logs",
"application"
],
"editable": false,
"schemaVersion": 39,
"version": 1,
"refresh": "",
"time": {
"from": "now-1h",
"to": "now"
},
"timezone": "browser",
"templating": {
"list": [
{
"name": "source",
"label": "Source",
"type": "custom",
"query": "Keycloak : keycloak/keycloak,LibreChat : librechat/librechat,Classic Models API : classic-models/classic-models-api,Kong gateways : (catalog|customers|payments|sales|workforce|llm)/proxy",
"multi": true,
"includeAll": true,
"allValue": "keycloak/keycloak|librechat/librechat|classic-models/classic-models-api|(catalog|customers|payments|sales|workforce|llm)/proxy",
"current": {
"text": [
"All"
],
"value": [
"$__all"
]
}
},
{
"name": "level",
"label": "Level",
"type": "custom",
"query": "error,warn,notice,info,debug,crit",
"multi": true,
"includeAll": true,
"allValue": "error|warn|notice|info|debug|crit|critical|fatal|emerg|alert",
"current": {
"text": [
"error",
"warn"
],
"value": [
"error",
"warn"
]
}
},
{
"name": "search",
"label": "Search",
"type": "textbox",
"query": "",
"current": {
"text": "",
"value": ""
},
"description": "Case-insensitive regex over the whole line. Empty matches everything."
}
]
},
"panels": [
{
"id": 1,
"type": "stat",
"title": "Errors",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 0,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__range])) or vector(0))",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"description": "Independent of the Level selector, so it still answers \"is anything broken here?\" while you are filtered on something else."
},
{
"id": 2,
"type": "stat",
"title": "Warnings",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 6,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level = \"warn\" [$__range])) or vector(0))",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "thresholds"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
},
{
"color": "orange",
"value": 1
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 3,
"type": "stat",
"title": "Lines matching",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "(sum(count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0)) + (sum(count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__range])) or vector(0))",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "blue"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 4,
"type": "stat",
"title": "Sources active",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 0
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "count(sum by (job) (count_over_time({job=~\"${source:regex}\"} |~ `(?i)$search` [$__range]))) or vector(0)",
"queryType": "instant"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"decimals": 0,
"mappings": [],
"color": {
"mode": "fixed",
"fixedColor": "purple"
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "purple",
"value": null
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
}
},
{
"id": 5,
"type": "timeseries",
"title": "Volume by level",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 4
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
"legendFormat": "{{level}}",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
"legendFormat": "{{level}}",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
"legendFormat": "{{level}}",
"queryType": "range"
},
{
"refId": "D",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (level) (count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\" [$__auto]))",
"legendFormat": "{{level}}",
"queryType": "range"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"mappings": [],
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "bars",
"lineWidth": 1,
"fillOpacity": 80,
"showPoints": "never",
"barAlignment": 0,
"stacking": {
"mode": "normal",
"group": "A"
}
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "critical"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "dark-red"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "error"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "red"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "warn"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "orange"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "info"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "green"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "debug"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "blue"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "unknown"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "notice"
},
"properties": [
{
"id": "color",
"value": {
"mode": "fixed",
"fixedColor": "text"
}
}
]
}
]
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"sum"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "Levels as each runtime states them, lowercased so nginx's `notice` and Keycloak's `INFO` land on the same scale."
},
{
"id": 6,
"type": "timeseries",
"title": "Errors by source",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
},
{
"refId": "D",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "sum by (job) (count_over_time({job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"error|crit|critical|fatal|emerg|alert\" [$__auto]))",
"legendFormat": "{{job}}",
"queryType": "range"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0,
"mappings": [],
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "bars",
"lineWidth": 1,
"fillOpacity": 80,
"showPoints": "never",
"barAlignment": 0,
"stacking": {
"mode": "normal",
"group": "A"
}
}
},
"overrides": []
},
"options": {
"legend": {
"showLegend": true,
"displayMode": "table",
"placement": "bottom",
"calcs": [
"sum"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"description": "Empty is the healthy state."
},
{
"id": 7,
"type": "table",
"title": "Application log lines",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 14,
"w": 24,
"x": 0,
"y": 12
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range",
"maxLines": 1000
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range",
"maxLines": 1000
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range",
"maxLines": 1000
},
{
"refId": "D",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range",
"maxLines": 1000
}
],
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "merge",
"options": {}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|job|level|msg|logger)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"job": 1,
"level": 2,
"msg": 3,
"logger": 4,
"body": 5
},
"renameByName": {
"timestamp": "Time",
"body": "Raw line",
"job": "Source",
"level": "Level",
"msg": "Message",
"logger": "Logger"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Source"
},
"properties": [
{
"id": "custom.width",
"value": 210
}
]
},
{
"matcher": {
"id": "byName",
"options": "Level"
},
"properties": [
{
"id": "custom.width",
"value": 90
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^(fatal|FATAL|critical|CRITICAL)$",
"result": {
"color": "dark-red",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(error|ERROR|Error)$",
"result": {
"color": "red",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(warn|WARN|warning|WARNING)$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(info|INFO|Info)$",
"result": {
"color": "green",
"index": 3
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(debug|DEBUG|Debug|trace|TRACE)$",
"result": {
"color": "blue",
"index": 4
}
}
},
{
"type": "regex",
"options": {
"pattern": "^unknown$",
"result": {
"color": "text",
"index": 5
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Logger"
},
"properties": [
{
"id": "custom.width",
"value": 240
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw line"
},
"properties": [
{
"id": "custom.width",
"value": 240
}
]
}
]
},
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "Each runtime parsed by its own format — Keycloak JSON, LibreChat colour-coded text, Django JSON, Kong's nginx error log — normalised to level and message. Logger is populated for Keycloak only, the one source that names the emitting class."
},
{
"id": 8,
"type": "logs",
"title": "Application logs as a stream",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 26
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"keycloak/keycloak\"} |~ `(?i)$search` | json level=\"level\", logger=\"loggerName\", msg=\"message\" | logger !~ \"org.keycloak.http.access-log|org.keycloak.events.*\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range"
},
{
"refId": "B",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"librechat/librechat\"} |~ `(?i)$search` | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<_> <_> <level>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range"
},
{
"refId": "C",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=\"classic-models/classic-models-api\"} !~ `^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+ - - \\[` |~ `(?i)$search` | json level=\"levelname\", msg=\"message\" | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range"
},
{
"refId": "D",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"${source:regex}\", job=~\"(catalog|customers|payments|sales|workforce|llm)/proxy\"} |~ `^[0-9]{4}/[0-9]{2}/[0-9]{2} ` |~ `(?i)$search` | pattern `<_> <_> [<level>] <_>: <msg>` | label_format level=`{{ lower .level }}` | level =~ \"${level:regex}\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": true,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Same result set, with ANSI colour and inline detail expansion."
}
]
}
k8s/grafana/dashboards/obs-audit-logs.json
+1325
View File
File diff suppressed because it is too large Load Diff
k8s/grafana/dashboards/obs-classic-models-api.json
+272 -19
View File
@@ -1384,9 +1384,9 @@
"description": "Namespace-wide so the datastore's failures show up next to the application's."
},
{
"id": 20,
"type": "logs",
"title": "Recent errors and warnings",
"id": 25,
"type": "table",
"title": "Access log",
"datasource": {
"type": "loki",
"uid": "loki"
@@ -1404,21 +1404,274 @@
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"classic-models\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
"expr": "{namespace=\"classic-models\", container=\"classic-models-api\"} | pattern `<client> - - [<ts>] \"<method> <uri> <proto>\" <status> <size> <_> \"<agent>\"`",
"queryType": "range",
"maxLines": 500
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|status|method|uri|size|client|agent)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"status": 1,
"method": 2,
"uri": 3,
"size": 4,
"client": 5,
"agent": 6,
"body": 7
},
"renameByName": {
"status": "Status",
"method": "Method",
"uri": "URI",
"size": "Bytes",
"client": "Client",
"agent": "User agent",
"body": "Raw record",
"timestamp": "Time"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Status"
},
"properties": [
{
"id": "custom.width",
"value": 80
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^2..$",
"result": {
"color": "green",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^3..$",
"result": {
"color": "blue",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^4..$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^5..$",
"result": {
"color": "red",
"index": 3
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Method"
},
"properties": [
{
"id": "custom.width",
"value": 80
}
]
},
{
"matcher": {
"id": "byName",
"options": "URI"
},
"properties": [
{
"id": "custom.width",
"value": 300
}
]
},
{
"matcher": {
"id": "byName",
"options": "Bytes"
},
"properties": [
{
"id": "custom.width",
"value": 90
}
]
},
{
"matcher": {
"id": "byName",
"options": "Client"
},
"properties": [
{
"id": "custom.width",
"value": 130
}
]
},
{
"matcher": {
"id": "byName",
"options": "User agent"
},
"properties": [
{
"id": "custom.width",
"value": 200
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw record"
},
"properties": [
{
"id": "custom.width",
"value": 260
}
]
}
]
},
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "gunicorn writes NCSA access lines, not JSON, so these are split by pattern. Lines that do not match — application JSON logs — are kept with empty columns rather than dropped."
},
{
"id": 26,
"type": "row",
"title": "Raw log lines",
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 58
},
"panels": [
{
"id": 20,
"type": "logs",
"title": "Raw log lines",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 59
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"classic-models\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
}
]
},
{
"id": 21,
@@ -1429,7 +1682,7 @@
"h": 1,
"w": 24,
"x": 0,
"y": 58
"y": 59
},
"panels": []
},
@@ -1445,7 +1698,7 @@
"h": 7,
"w": 8,
"x": 0,
"y": 59
"y": 60
},
"targets": [
{
@@ -1511,7 +1764,7 @@
"h": 7,
"w": 8,
"x": 8,
"y": 59
"y": 60
},
"targets": [
{
@@ -1576,7 +1829,7 @@
"h": 7,
"w": 8,
"x": 16,
"y": 59
"y": 60
},
"targets": [
{
k8s/grafana/dashboards/obs-distributed-traces.json
+451
View File
@@ -0,0 +1,451 @@
{
"uid": "obs-distributed-traces",
"title": "Distributed Traces",
"description": "Follow one transaction end to end: pick it, read every span in order, see which tokens were issued, and read what was exchanged at each hop.",
"tags": [
"observability",
"tracing",
"tempo"
],
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"refresh": "",
"time": {
"from": "now-1h",
"to": "now"
},
"editable": true,
"graphTooltip": 0,
"templating": {
"list": [
{
"name": "traceFilter",
"label": "Show",
"type": "custom",
"query": "LibreChat transactions : {trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"},Reaching a Kong gateway : {trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"},All LibreChat traces (incl. /metrics scrapes) : {trace:rootService=\"librechat\"},Everything (very noisy) : {}",
"options": [
{
"text": "LibreChat transactions",
"value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}",
"selected": true
},
{
"text": "Reaching a Kong gateway",
"value": "{trace:rootService=\"librechat\"} && {name=~\"kong\\\\..*\"}",
"selected": false
},
{
"text": "All LibreChat traces (incl. /metrics scrapes)",
"value": "{trace:rootService=\"librechat\"}",
"selected": false
},
{
"text": "Everything (very noisy)",
"value": "{}",
"selected": false
}
],
"current": {
"text": "LibreChat transactions",
"value": "{trace:rootService=\"librechat\"} && {resource.service.name!=\"librechat\"}"
},
"includeAll": false,
"multi": false,
"description": "Traces started BY LibreChat and leaving it. Rooting on LibreChat is what makes this a user-facing transaction rather than infrastructure chatter; requiring a second service is what drops Prometheus scraping LibreChat's own /metrics, which otherwise fills the list."
},
{
"name": "traceId",
"label": "Trace ID",
"type": "textbox",
"query": "",
"current": {
"text": "",
"value": ""
},
"description": "Set by clicking a Trace ID above, or paste one."
},
{
"name": "userId",
"label": "User ID",
"type": "textbox",
"query": "",
"current": {
"text": "",
"value": ""
},
"description": "Set by clicking an identity in the panel below. Empty means the Keycloak panel is scoped only by time."
}
]
},
"panels": [
{
"type": "row",
"title": "1 — Pick a transaction",
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"collapsed": false,
"panels": []
},
{
"type": "table",
"title": "Transactions",
"description": "Traces LibreChat started that reached another service — the user-facing transactions.\n\nCounting spans was the previous approach and it let LibreChat's own /metrics scrapes through: with 50 of them in six hours they crowded out the two real conversations. Rooting on the service that starts a transaction is both simpler and stricter.\n\nThe trade: anything not entering through LibreChat is invisible here — a direct call to a gateway, or a curl against an API. Use 'Everything' for those.\n\nClick a Trace ID to load it below.",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 1
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"queryType": "traceql",
"query": "$traceFilter",
"tableType": "traces",
"limit": 50
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"filterable": true
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "traceID"
},
"properties": [
{
"id": "links",
"value": [
{
"title": "Open this trace below",
"url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${__value.raw}"
}
]
}
]
}
]
},
"options": {
"showHeader": true,
"sortBy": [
{
"displayName": "Start time",
"desc": true
}
]
}
},
{
"type": "row",
"title": "2 — Every span of the trace, in order",
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 9
},
"collapsed": false,
"panels": []
},
{
"type": "traces",
"title": "Trace $traceId",
"description": "Grafana's own trace view: every span nested in order across LibreChat, Keycloak, the Kong gateways and the backend. Click a span to expand its attributes; from a span, 'Logs for this span' jumps to Loki.\n\nThe framework plumbing — Express middleware, mongoose/mongodb, kong.dns, raw SQL — is most of the span count and cannot be hidden from here: this panel's span-name filter is an exact string comparison on operationName, so a pattern matches nothing and every span passes. Only the tag filters take a regex, and a span's name is not a tag. Use the panel's own 'Critical path' button to collapse the view to what determines the duration.\n\nRemoving those spans for real means turning the instrumentations off at the source.",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"gridPos": {
"h": 22,
"w": 24,
"x": 0,
"y": 10
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "tempo",
"uid": "tempo"
},
"queryType": "traceql",
"query": "$traceId",
"limit": 300
}
],
"options": {}
},
{
"type": "row",
"title": "3 — Identity: what was issued, to whom, for what",
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 32
},
"collapsed": false,
"panels": []
},
{
"type": "table",
"title": "Identities authenticated in this trace",
"description": "Who Kong authenticated at each hop of the selected trace, read from the audit records' authenticated_entity.id. A transaction usually carries more than one: LibreChat's own identity on the llm hop, the end user's on the domain hops.\n\nClick a User ID to scope the Keycloak panel below to it. That click is the join Kong does not make for us — its openid-connect plugin calls Keycloak through its own HTTP client, which Kong's propagation module does not touch, so those token exchanges carry their own trace id. The user id is the same on both sides, which is what makes the join possible.",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 33
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"editorMode": "code",
"queryType": "range",
"expr": "{job=\"kong-audit\"} |= \"$traceId\" | json authuser=\"authenticated_entity.id\", svc=\"service.name\", uri=\"request.uri\", status=\"response.status\""
}
],
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "organize",
"options": {
"excludeByName": {
"labels": true,
"Line": true,
"tsNs": true,
"id": true
},
"indexByName": {
"Time": 0,
"authuser": 1,
"svc": 2,
"uri": 3,
"status": 4
},
"renameByName": {
"authuser": "User ID",
"svc": "Gateway service",
"uri": "Path",
"status": "Status"
}
}
},
{
"id": "groupBy",
"options": {
"fields": {
"User ID": {
"aggregations": [],
"operation": "groupby"
},
"Gateway service": {
"aggregations": [],
"operation": "groupby"
},
"Path": {
"aggregations": [],
"operation": "groupby"
},
"Status": {
"aggregations": [],
"operation": "groupby"
}
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"filterable": true
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "User ID"
},
"properties": [
{
"id": "links",
"value": [
{
"title": "Scope the Keycloak panel to this identity",
"url": "d/obs-distributed-traces/distributed-traces?${__url_time_range}&var-traceId=${traceId}&var-userId=${__value.raw}"
}
]
}
]
}
]
},
"options": {
"showHeader": true
}
},
{
"type": "table",
"title": "Keycloak identity events",
"description": "Parsed from Keycloak's own event log: token exchanges, logins and code-to-token swaps, with the audience minted, the requesting client, the end user and the grant type. This is where the OIDC substance lives — the Authorization header is REDACTED in the Kong audit records, so claims cannot be read from there, and the raw token is deliberately not logged anywhere.\n\nScoped by USER, not by trace, because Kong does not propagate the trace when its openid-connect plugin calls Keycloak — verified: a request traced end to end produced a TOKEN_EXCHANGE that Keycloak recorded under its own trace id. Click a User ID in the panel above to fill it in; with it empty this shows every identity event in the time range.",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 41
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"editorMode": "code",
"queryType": "range",
"expr": "{job=\"keycloak/keycloak\"} |= \"org.keycloak.events\" |= \"$userId\" | regexp `type=\\\\\"(?P<event>[^\\\\\"]*)\\\\\"` | regexp `clientId=\\\\\"(?P<client>[^\\\\\"]*)\\\\\"` | regexp `audience=\\\\\"(?P<audience>[^\\\\\"]*)\\\\\"` | regexp `username=\\\\\"(?P<user>[^\\\\\"]*)\\\\\"` | regexp `grant_type=\\\\\"(?P<grant>[^\\\\\"]*)\\\\\"` | regexp `scope=\\\\\"(?P<scope>[^\\\\\"]*)\\\\\"`"
}
],
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "organize",
"options": {
"excludeByName": {
"labels": true,
"Line": true,
"tsNs": true,
"id": true,
"Time": false
},
"indexByName": {
"Time": 0,
"event": 1,
"client": 2,
"audience": 3,
"user": 4,
"grant": 5,
"scope": 6
},
"renameByName": {
"event": "Event",
"client": "Client",
"audience": "Audience",
"user": "User",
"grant": "Grant type",
"scope": "Scopes"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"filterable": true
}
},
"overrides": []
},
"options": {
"showHeader": true,
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
}
},
{
"type": "row",
"title": "4 — What was actually exchanged",
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 52
},
"collapsed": false,
"panels": []
},
{
"type": "logs",
"title": "Logs of trace $traceId",
"description": "Every log line carrying this trace id. The Kong audit records are the substance: request headers, response status, per-stage latencies, and on the domain gateways request_body and response_body — so the MCP tools/call payload and its result are readable in full. Expand a line to read it as fields.\n\nOne gap, and it is a capture gap rather than a display one: the llm gateway has no body capture, so LLM prompts and completions are not recorded.",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 14,
"w": 24,
"x": 0,
"y": 53
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{job=~\"kong-audit|classic-models/classic-models-api|librechat/librechat|keycloak/keycloak\"} |= \"$traceId\""
}
],
"options": {
"showTime": true,
"wrapLogMessage": true,
"sortOrder": "Ascending",
"enableLogDetails": true,
"prettifyLogMessage": true
}
}
]
}
k8s/grafana/dashboards/obs-keycloak.json
+242 -19
View File
@@ -1387,9 +1387,9 @@
"description": "Namespace-wide so the datastore's failures show up next to the application's."
},
{
"id": 21,
"type": "logs",
"title": "Recent errors and warnings",
"id": 26,
"type": "table",
"title": "Errors and warnings",
"datasource": {
"type": "loki",
"uid": "loki"
@@ -1407,21 +1407,244 @@
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"keycloak\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
"expr": "{namespace=\"keycloak\", container=\"keycloak\"} | json | detected_level =~ \"error|critical|warn\"",
"queryType": "range",
"maxLines": 500
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|level|loggerName|message|threadName)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"level": 1,
"loggerName": 2,
"message": 3,
"threadName": 4,
"body": 5
},
"renameByName": {
"level": "Level",
"loggerName": "Logger",
"message": "Message",
"threadName": "Thread",
"body": "Raw record",
"timestamp": "Time"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Level"
},
"properties": [
{
"id": "custom.width",
"value": 90
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
"result": {
"color": "dark-red",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(error|ERROR|Error)$",
"result": {
"color": "red",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(info|INFO|Info)$",
"result": {
"color": "green",
"index": 3
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(debug|DEBUG|Debug)$",
"result": {
"color": "blue",
"index": 4
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Logger"
},
"properties": [
{
"id": "custom.width",
"value": 280
}
]
},
{
"matcher": {
"id": "byName",
"options": "Thread"
},
"properties": [
{
"id": "custom.width",
"value": 160
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw record"
},
"properties": [
{
"id": "custom.width",
"value": 260
}
]
}
]
},
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "Keycloak logs JSON, so the fields come straight from `| json`. Scoped to the Keycloak container: PostgreSQL in the same namespace is not JSON and would parse to empty columns."
},
{
"id": 27,
"type": "row",
"title": "Raw log lines",
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 60
},
"panels": [
{
"id": 21,
"type": "logs",
"title": "Raw log lines",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 61
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"keycloak\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
}
]
},
{
"id": 22,
@@ -1432,7 +1655,7 @@
"h": 1,
"w": 24,
"x": 0,
"y": 60
"y": 61
},
"panels": []
},
@@ -1448,7 +1671,7 @@
"h": 7,
"w": 8,
"x": 0,
"y": 61
"y": 62
},
"targets": [
{
@@ -1514,7 +1737,7 @@
"h": 7,
"w": 8,
"x": 8,
"y": 61
"y": 62
},
"targets": [
{
@@ -1579,7 +1802,7 @@
"h": 7,
"w": 8,
"x": 16,
"y": 61
"y": 62
},
"targets": [
{
k8s/grafana/dashboards/obs-kong-gateways.json
+1714
View File
File diff suppressed because it is too large Load Diff
k8s/grafana/dashboards/obs-librechat.json
+214 -19
View File
@@ -1522,9 +1522,9 @@
"description": "Namespace-wide so the datastore's failures show up next to the application's."
},
{
"id": 22,
"type": "logs",
"title": "Recent errors and warnings",
"id": 27,
"type": "table",
"title": "Errors and warnings",
"datasource": {
"type": "loki",
"uid": "loki"
@@ -1542,21 +1542,216 @@
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"librechat\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
"expr": "{namespace=\"librechat\", container=\"librechat\"} | detected_level =~ \"error|critical|warn\" | line_format `{{ regexReplaceAll \"\\\\x1b\\\\[[0-9;]*m\" __line__ \"\" }}` | pattern `<date> <time> <lvl>: <msg>`",
"queryType": "range",
"maxLines": 500
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|lvl|msg)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"lvl": 1,
"msg": 2,
"body": 3
},
"renameByName": {
"lvl": "Level",
"msg": "Message",
"body": "Raw record",
"timestamp": "Time"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Level"
},
"properties": [
{
"id": "custom.width",
"value": 90
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
"result": {
"color": "dark-red",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(error|ERROR|Error)$",
"result": {
"color": "red",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(info|INFO|Info)$",
"result": {
"color": "green",
"index": 3
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(debug|DEBUG|Debug)$",
"result": {
"color": "blue",
"index": 4
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw record"
},
"properties": [
{
"id": "custom.width",
"value": 260
}
]
}
]
},
"description": "Click a line to expand it. Where the line carries a trace_id, the Loki datasource's derived field opens the matching trace in Tempo."
"options": {
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "LibreChat writes colour-coded text rather than JSON, so the ANSI escapes are stripped first and the remainder split by pattern. The Raw column therefore holds the cleaned line; the collapsed panel below keeps the original with its colours."
},
{
"id": 28,
"type": "row",
"title": "Raw log lines",
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 60
},
"panels": [
{
"id": 22,
"type": "logs",
"title": "Raw log lines",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 11,
"w": 24,
"x": 0,
"y": 61
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"librechat\"} | detected_level =~ \"error|critical|warn\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"showCommonLabels": false,
"wrapLogMessage": true,
"prettifyLogMessage": false,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
}
]
},
{
"id": 23,
@@ -1567,7 +1762,7 @@
"h": 1,
"w": 24,
"x": 0,
"y": 60
"y": 61
},
"panels": []
},
@@ -1583,7 +1778,7 @@
"h": 7,
"w": 8,
"x": 0,
"y": 61
"y": 62
},
"targets": [
{
@@ -1649,7 +1844,7 @@
"h": 7,
"w": 8,
"x": 8,
"y": 61
"y": 62
},
"targets": [
{
@@ -1714,7 +1909,7 @@
"h": 7,
"w": 8,
"x": 16,
"y": 61
"y": 62
},
"targets": [
{
k8s/grafana/dashboards/obs-stack-health.json
+250 -12
View File
@@ -400,9 +400,9 @@
"panels": []
},
{
"id": 10,
"type": "logs",
"title": "Recent stack errors",
"id": 11,
"type": "table",
"title": "Stack errors and warnings",
"datasource": {
"type": "loki",
"uid": "loki"
@@ -420,18 +420,256 @@
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"observability\"} |~ \"(?i)(error|level=error|fatal)\"",
"queryType": "range"
"expr": "{namespace=\"observability\"} | logfmt level, msg, caller, logger, component_id | detected_level =~ \"error|critical|warn\"",
"queryType": "range",
"maxLines": 500
}
],
"transformations": [
{
"id": "extractFields",
"options": {
"source": "labels",
"replace": false
}
},
{
"id": "filterFieldsByName",
"options": {
"include": {
"pattern": "^(timestamp|body|container|level|msg|caller|component_id)$"
}
}
},
{
"id": "organize",
"options": {
"excludeByName": {},
"indexByName": {
"timestamp": 0,
"container": 1,
"level": 2,
"msg": 3,
"caller": 4,
"component_id": 5,
"body": 6
},
"renameByName": {
"container": "Component",
"level": "Level",
"msg": "Message",
"caller": "Caller",
"component_id": "Component id",
"body": "Raw record",
"timestamp": "Time"
}
}
}
],
"fieldConfig": {
"defaults": {
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": true,
"filterable": true
},
"mappings": []
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Component"
},
"properties": [
{
"id": "custom.width",
"value": 120
}
]
},
{
"matcher": {
"id": "byName",
"options": "Level"
},
"properties": [
{
"id": "custom.width",
"value": 90
},
{
"id": "mappings",
"value": [
{
"type": "regex",
"options": {
"pattern": "^(fatal|FATAL|Fatal|critical|CRITICAL|Critical)$",
"result": {
"color": "dark-red",
"index": 0
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(error|ERROR|Error)$",
"result": {
"color": "red",
"index": 1
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(warn|WARN|Warn|warning|WARNING|Warning)$",
"result": {
"color": "orange",
"index": 2
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(info|INFO|Info)$",
"result": {
"color": "green",
"index": 3
}
}
},
{
"type": "regex",
"options": {
"pattern": "^(debug|DEBUG|Debug)$",
"result": {
"color": "blue",
"index": 4
}
}
}
]
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-text"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "Caller"
},
"properties": [
{
"id": "custom.width",
"value": 200
}
]
},
{
"matcher": {
"id": "byName",
"options": "Component id"
},
"properties": [
{
"id": "custom.width",
"value": 200
}
]
},
{
"matcher": {
"id": "byName",
"options": "Raw record"
},
"properties": [
{
"id": "custom.width",
"value": 260
}
]
}
]
},
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
}
"showHeader": true,
"cellHeight": "sm",
"footer": {
"show": false,
"reducer": [
"sum"
],
"countRows": false,
"fields": ""
},
"sortBy": [
{
"displayName": "Time",
"desc": true
}
]
},
"description": "Prometheus, Loki, Tempo, Grafana and Alloy all write logfmt, so one selective parser covers the whole stack. Selective on purpose: a bare `| logfmt` would explode Loki's own query-metrics lines into dozens of columns."
},
{
"id": 12,
"type": "row",
"title": "Raw log lines",
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 26
},
"panels": [
{
"id": 10,
"type": "logs",
"title": "Raw log lines",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"h": 10,
"w": 24,
"x": 0,
"y": 27
},
"targets": [
{
"refId": "A",
"datasource": {
"type": "loki",
"uid": "loki"
},
"expr": "{namespace=\"observability\"} |~ \"(?i)(error|level=error|fatal)\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"sortOrder": "Descending",
"dedupStrategy": "none",
"enableLogDetails": true
},
"description": "Kept alongside the table: the logs panel renders ANSI colour, expands a record's detail inline, tails live, and turns a trace id into a link to Tempo — none of which a table can do. Collapsed so it stays out of the way until needed."
}
]
}
]
}
k8s/grafana/provisioning/datasources.yaml
+31 -8
View File
@@ -64,19 +64,42 @@ datasources:
datasourceUid: prometheus
nodeGraph:
enabled: true
# Span -> logs by trace id rather than by service name.
#
# The service-name mapping this replaces resolved `service.name` to Loki's
# `app` label, which Alloy derives from app.kubernetes.io/name. That label
# exists for keycloak and classic-models-api but not for the six Kong
# gateways or librechat — and the gateways are precisely where the trace
# id does appear in the logs. The mapping pointed at the services where it
# could not work.
#
# A custom query on ${__trace.traceId} sidesteps the label entirely: every
# source that writes the id anywhere in the line is matched. The stream
# selector is spelled out rather than `.+` so the query stays bounded.
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: -5m
spanEndTimeShift: 5m
filterByTraceID: true
spanStartTimeShift: -1h
spanEndTimeShift: 1h
filterByTraceID: false
filterBySpanID: false
tags:
- key: service.name
value: app
customQuery: true
query: '{job=~"kong-audit|keycloak/keycloak|classic-models/classic-models-api|librechat/librechat"} |= "$${__trace.traceId}"'
# Span -> metrics against the span metrics Tempo's own metrics_generator
# feeds into Prometheus. Without explicit queries this block renders a
# bare datasource link and nothing else, which is what it did before.
# $$__tags expands to the tag mapping below, so each query is scoped to
# the span's own service.
tracesToMetrics:
datasourceUid: prometheus
spanStartTimeShift: -5m
spanEndTimeShift: 5m
spanStartTimeShift: -30m
spanEndTimeShift: 30m
tags:
- key: service.name
value: service
queries:
- name: 'Request rate'
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags}[$$__rate_interval]))'
- name: 'Error rate'
query: 'sum by (span_name) (rate(traces_spanmetrics_calls_total{$$__tags,status_code="STATUS_CODE_ERROR"}[$$__rate_interval]))'
- name: 'Latency p95'
query: 'histogram_quantile(0.95, sum by (span_name, le) (rate(traces_spanmetrics_latency_bucket{$$__tags}[$$__rate_interval])))'
k8s/kustomization.yaml
+4 -1
View File
@@ -4,13 +4,16 @@ namespace: observability
resources:
- namespace.yaml
- secret.yaml
# observability-secrets now comes from Vault through External Secrets;
# k8s/secret.yaml is no longer applied and can be deleted.
- secrets
- prometheus
- loki
- tempo
- alloy
- grafana
- grafana/dashboards
- mcp-grafana
- ingress
labels:
k8s/mcp-grafana/deployment.yaml
+96
View File
@@ -0,0 +1,96 @@
# Grafana's own MCP server: it exposes Prometheus, Loki and Tempo as tools an
# LLM can call, so a question like "what failed in the last transaction" is
# answered by querying, rather than by building another panel.
#
# Works against OSS Grafana 9.0+. The Cloud-only tools — Incidents, OnCall,
# Sift, Agent Observability, Assistant — are simply absent here; the
# datasource, dashboard and alerting tools are the ones that matter.
#
# Reachable in-cluster only for now, with no OIDC and no audit trail. The
# intended end state is the same treatment as the five domain MCP servers:
# behind a Kong gateway with openid-connect, consumer groups and http-log.
# Until then, anything that can reach this Service can read all the telemetry.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mcp-grafana
labels:
app.kubernetes.io/name: mcp-grafana
app.kubernetes.io/component: mcp
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: mcp-grafana
template:
metadata:
labels:
app.kubernetes.io/name: mcp-grafana
app.kubernetes.io/component: mcp
spec:
containers:
- name: mcp-grafana
image: mcp/grafana:latest
args:
# Streamable HTTP rather than stdio: this runs as a service other
# pods connect to, not as a subprocess of one client.
- "-t"
- "streamable-http"
- "--log-level"
- "info"
# DNS-rebinding protection: the server rejects any Host header not
# on this list, defaulting to loopback only — which is why the first
# in-cluster probe got "forbidden: host not allowed". Name the
# Service instead of disabling validation with "*": the point of the
# check is that only callers using the expected name get through.
- "--allowed-hosts"
- "mcp-grafana:8000,mcp-grafana.observability.svc.cluster.local:8000"
# 65 tools by default, 75 KiB of definitions — and an MCP client
# sends the whole tool list on every turn. Measured: that alone was
# ~19k prompt tokens per request, which is what made a three-turn
# conversation trip Kong's 50k-tokens-per-minute AI rate limit and
# fail with 429 before reaching the provider.
#
# Kept: the datasources this stack actually has, plus search to find
# things by name. Dropped: everything Cloud-only here (incident,
# oncall, sift, asserts, pyroscope), every datasource that does not
# exist (athena, clickhouse, cloudwatch, influxdb, elasticsearch,
# graphite, quickwit, snowflake), and the Grafana-administration
# surface (provisioning, plugin, api, config, folder, annotations,
# snapshot, rendering, navigation).
- "--enabled-tools"
- "search,datasource,prometheus,loki"
# The service account is Viewer, so writes would fail anyway —
# this removes their definitions from what is sent every turn.
- "--disable-write"
env:
- name: GRAFANA_URL
# In-cluster address, not the ingress: no TLS hop, no dependency
# on sttlab.local resolving inside the cluster.
value: http://grafana:3000
- name: GRAFANA_SERVICE_ACCOUNT_TOKEN
valueFrom:
secretKeyRef:
name: observability-secrets
key: GRAFANA_SERVICE_ACCOUNT_TOKEN
ports:
- name: http
containerPort: 8000
resources:
requests:
cpu: 20m
memory: 64Mi
limits:
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
# Numeric, because the image declares its user by name
# (USER mcp-grafana). Kubernetes cannot resolve a name to check it
# is not root, and refuses to start the container with
# runAsNonRoot: "image has non-numeric user". 1000/999 are what the
# image actually uses.
runAsUser: 1000
runAsGroup: 999
capabilities:
drop: ["ALL"]
k8s/mcp-grafana/kustomization.yaml
+5
View File
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
k8s/mcp-grafana/service.yaml
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: mcp-grafana
labels:
app.kubernetes.io/name: mcp-grafana
spec:
type: ClusterIP
selector:
app.kubernetes.io/name: mcp-grafana
ports:
- name: http
port: 8000
targetPort: http
k8s/secret.example.yaml
+14 -11
View File
@@ -1,11 +1,14 @@
# Template: copy to secret.yaml (gitignored) and fill in the values.
# make secret
apiVersion: v1
kind: Secret
metadata:
name: observability-secrets
namespace: observability
type: Opaque
stringData:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: <CHANGE_ME>
# NO LONGER APPLIED. observability-secrets now comes from Vault through
# External Secrets — see k8s/secrets/. This file is kept only to document
# which keys the stack expects and where they live.
#
# Vault path: secrets/observability/grafana
# GF_SECURITY_ADMIN_USER
# GF_SECURITY_ADMIN_PASSWORD
# GRAFANA_SERVICE_ACCOUNT_TOKEN token of the 'mcp-grafana' service
# account, Viewer role. Grafana returns it once, at creation:
# POST /api/serviceaccounts {"name":"mcp-grafana","role":"Viewer"}
# POST /api/serviceaccounts/<id>/tokens {"name":"mcp-grafana-token"}
#
# The ExternalSecret pulls the whole path, so adding a key in Vault reaches
# the cluster without editing anything here.
k8s/secrets/clustersecretstore.yaml
+35
View File
@@ -0,0 +1,35 @@
# Reads secrets/observability/* from Vault. Deliberately not reusing the
# vault-kong-v2 store: its AppRole carries the kong-v2-readonly policy, scoped
# to secrets/data/kong-v2/*, so it simply cannot see this path — verified, a
# probe ExternalSecret against it failed with "could not get secret data from
# provider". Widening that policy would have given the Kong reader access to
# observability's secrets for no reason.
#
# The AppRole credentials are NOT in git. Created once against Vault:
# vault policy write observability-readonly (read on secrets/{data,metadata}/observability/*)
# vault write auth/approle/role/observability-eso token_policies=observability-readonly
# vault read auth/approle/role/observability-eso/role-id
# vault write -f auth/approle/role/observability-eso/secret-id
# kubectl -n external-secrets create secret generic vault-approle-observability \
# --from-literal=roleId=<...> --from-literal=secretId=<...>
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: vault-observability
spec:
provider:
vault:
server: https://vault.sttlab.eu
path: secrets
version: v2
auth:
appRole:
path: approle
roleRef:
name: vault-approle-observability
namespace: external-secrets
key: roleId
secretRef:
name: vault-approle-observability
namespace: external-secrets
key: secretId
k8s/secrets/externalsecret.yaml
+23
View File
@@ -0,0 +1,23 @@
# Replaces the gitignored k8s/secret.yaml: the Grafana admin credentials and
# the service account token the MCP server authenticates with now come from
# Vault, so nothing sensitive has to exist on a laptop to deploy this stack.
#
# The target keeps the name every consumer already uses, so neither the Grafana
# Deployment nor mcp-grafana changes.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: observability-secrets
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: vault-observability
target:
name: observability-secrets
creationPolicy: Owner
dataFrom:
# Whole-secret pull rather than key-by-key: adding a key in Vault then
# reaches the cluster without editing this file.
- extract:
key: observability/grafana
k8s/secrets/kustomization.yaml
+5
View File
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- clustersecretstore.yaml
- externalsecret.yaml