Split log search into access, application and audit dashboards

One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.

Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.

The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.

Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.

Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-25 18:48:11 +02:00
parent 40ffac4acf
commit eb35419ad0
9 changed files with 2696 additions and 320 deletions
+9 -4
View File
@@ -46,7 +46,8 @@ help: ## Show this help
.PHONY: deploy
deploy: check-context secret ## Deploy the whole stack (runtimes + dashboards)
$(KUBECTL) apply -k $(K8S)
# --server-side for the same reason as the dashboards target below.
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts
$(MAKE) --no-print-directory wait
.PHONY: deploy-runtime
@@ -89,7 +90,11 @@ secret: ## Generate k8s/secret.yaml (random admin password) if absent
.PHONY: dashboards
dashboards: check-context dashboards-check ## Apply dashboards only (no Grafana restart)
$(KUBECTL) apply -k $(DASHBOARDS)
# Server-side apply: client-side apply stores a full copy of the object in the
# kubectl.kubernetes.io/last-applied-configuration annotation, which is capped at
# 256KB. The dashboards passed that in aggregate. Server-side apply tracks field
# ownership in metadata instead, so the ceiling becomes etcd's ~1MB per object.
$(KUBECTL) apply -k $(DASHBOARDS) --server-side --force-conflicts
printf "$(C_OK)grafana-dashboards ConfigMap updated$(C_OFF) -- visible in Grafana within ~70s\n"
printf " (kubelet volume sync ~60s + provisioner poll 10s, no restart).\n"
printf " For immediate effect: $(C_WARN)make restart-grafana$(C_OFF)\n"
@@ -310,7 +315,7 @@ _flush:
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=0
$(KUBECTL) -n $(NS) wait --for=delete pod -l app=$(COMP) --timeout=120s 2>/dev/null || true
$(KUBECTL) -n $(NS) delete pvc $(COMP)-data --ignore-not-found --wait=true
$(KUBECTL) apply -k $(K8S)/$(COMP)
$(KUBECTL) apply -k $(K8S)/$(COMP) --server-side --force-conflicts
$(KUBECTL) -n $(NS) scale deployment/$(COMP) --replicas=1
##@ Validation and teardown
@@ -326,7 +331,7 @@ diff: check-context ## Diff the manifests against the cluster state
.PHONY: validate
validate: dashboards-check ## Validate the manifests (build + server dry-run)
kustomize build $(K8S) >/dev/null && printf "$(C_OK)kustomize build OK$(C_OFF)\n"
$(KUBECTL) apply -k $(K8S) --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
$(KUBECTL) apply -k $(K8S) --server-side --force-conflicts --dry-run=server >/dev/null && printf "$(C_OK)server dry-run OK$(C_OFF)\n"
.PHONY: undeploy
undeploy: ## DESTRUCTIVE: delete the stack, PVCs kept (CONFIRM=yes)