Stéphane Tailland
eb35419ad0
One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.
Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.
The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.
Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.
Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
36 lines
1.2 KiB
YAML
36 lines
1.2 KiB
YAML
# Grafana dashboards -- declarative provisioning.
|
|
#
|
|
# Rules:
|
|
# * one .json file per dashboard, named after its UID;
|
|
# * every JSON carries a STABLE "uid" field (never regenerated): that is what
|
|
# keeps URLs, cross-links and alerts alive across redeployments;
|
|
# * `disableNameSuffixHash: true` -> the ConfigMap always keeps the same name,
|
|
# so `kubectl apply` updates it in place and Grafana reloads WITHOUT
|
|
# restarting the pod, in about 70s (see dashboard-providers.yaml).
|
|
#
|
|
# Adding a dashboard = drop the .json here and add it to `files:` below.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: observability
|
|
|
|
configMapGenerator:
|
|
- name: grafana-dashboards
|
|
files:
|
|
# >>> dashboards (generated by `make dashboards-sync`) >>>
|
|
- obs-access-logs.json
|
|
- obs-application-logs.json
|
|
- obs-audit-logs.json
|
|
- obs-classic-models-api.json
|
|
- obs-keycloak.json
|
|
- obs-kong-gateways.json
|
|
- obs-librechat.json
|
|
- obs-stack-health.json
|
|
# <<< dashboards <<<
|
|
|
|
generatorOptions:
|
|
disableNameSuffixHash: true
|
|
labels:
|
|
app.kubernetes.io/name: grafana
|
|
app.kubernetes.io/component: dashboards
|
|
app.kubernetes.io/part-of: observability
|