Split log search into access, application and audit dashboards

One consolidated search compromised on all three questions at once. It also
had a real defect: the Source selector carried `.+` as its All value, so
choosing "All" widened the query back to every stream in the cluster and the
scope preset only ever populated the picker. MongoDB and cert-manager showed up
in what was meant to be an application view.

Each query now carries both `job=~"${source:regex}"` and a literal job matcher,
so the selector actually restricts rather than widens.

The three kinds are subtracted from each other rather than merely filtered:
application logs exclude Keycloak's access-log and events loggers by name, the
Django container's NCSA lines by shape, and the Kong containers' stray access
lines by requiring the nginx date prefix.

Levels are read from the line rather than taken from Loki's detected_level.
That inference marks 78% of the Kong gateway output `unknown` even though every
one of those lines states [notice] or [warn], so filtering on it hid most of
the gateway's own logs. Each parser extracts the level and lowercases it, which
puts nginx's notice, Keycloak's INFO and LibreChat's info on one scale — and
removes the need for an `unknown` entry in the selector.

Applies move to server-side. Client-side apply stores a full copy of each
object in the last-applied-configuration annotation, capped at 256KB, and the
dashboard ConfigMap passed that at eight dashboards; the failure surfaced as
"metadata.annotations: Too long". Server-side apply tracks field ownership in
metadata instead, moving the ceiling to etcd's ~1MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -132,6 +132,13 @@ forces a pod restart when they change.
 | `tempo-data` | Tempo | 10Gi | WAL + trace blocks | 168 h |
 | `grafana-data` | Grafana | 5Gi | SQLite database (users, prefs) | — |
 
+> **Applies are server-side.** Client-side `kubectl apply` stores a full copy of
+> each object in the `kubectl.kubernetes.io/last-applied-configuration`
+> annotation, capped at 256KB; the dashboard ConfigMap passed that in aggregate
+> at eight dashboards. `--server-side` tracks field ownership in metadata
+> instead, which moves the ceiling to etcd's ~1MB per object. Splitting into one
+> ConfigMap per dashboard is the next step if that is ever reached.
+
 StorageClass `local-path` (the cluster default), `ReadWriteOnce`. The Deployments
 use `strategy: Recreate`: with an RWO volume the old pod must release it before
 the new one can start.
@@ -206,33 +213,45 @@ Each application dashboard carries the same four sections: an overview of RED
 indicators, HTTP traffic, a runtime section specific to the stack in use, and a
 **Logs** row, followed by container resources.
 
-### Consolidated log search
+### Log dashboards, split by kind
 
-`obs-log-search` searches every source at once, and is deliberately separate
-from the per-application dashboards: those answer "how is this component
-behaving", this one answers "where did that string come from".
+Logs are searched from three dashboards rather than one, because the three
+questions have nothing in common and a single view compromised on all of them.
 
-Four controls: a **scope** preset, a **source** picker chained to it, a
-**level** filter, and a free-text **search** box. The search term is injected as
-a case-insensitive line filter inside LogQL backticks — a raw string, so a quote
-typed in the box cannot break the query, and an empty box yields `(?i)`, a valid
-regex matching every line. That is what makes "no search" the natural default
-rather than a special case.
+| Dashboard | Question | Sources |
+|---|---|---|
+| `obs-access-logs` | Which HTTP requests came in? | Kong records, Keycloak access logger, Django gunicorn, ingress-nginx |
+| `obs-application-logs` | What is the code reporting, and what broke? | Keycloak, LibreChat, Django, the Kong dataplanes' nginx error log |
+| `obs-audit-logs` | Who authenticated, what changed, who called what? | Keycloak `org.keycloak.events`, Kong http-log records |
 
-Scope is a curated preset list rather than a free namespace picker, because the
-cluster carries 48 log sources and most are infrastructure noise. The presets
-are job regexes matched against the `<namespace>/<container>` label Alloy sets —
-except `kong-audit`, which Fluent Bit ships with no namespace at all, and which
-is why a single naive selector cannot span both collection paths.
+Each source keeps its own parser and they all emit the same field names, so the
+frames merge into one table with consistent columns. Access logs converge on
+method / path / status / client; application logs on level / message.
 
-Two things this dashboard deliberately does not do. It does not parse per
-format: its Message column holds the line as emitted, since the five sources
-agree on nothing and that is what the per-application dashboards are for. And
-its level filter is honest about being partial — around 40% of lines are
-`unknown` because access lines and Kong records carry no severity for Loki to
-infer, so for those the text is the only thing worth searching.
+Three things this split fixes that a single consolidated view could not:
 
-### Parsed log tables
+**The kinds are subtracted, not just filtered.** Application logs exclude
+Keycloak's `access-log` and `events` loggers by name, the Django container's
+NCSA lines by shape, and the Kong containers' stray access lines by requiring
+the nginx date prefix. Without that, access lines drown the signal.
+
+**Levels are read from the line, not inferred.** Loki's `detected_level` marks
+78% of the Kong gateway output `unknown` even though every one of those lines
+states `[notice]` or `[warn]`. Each parser extracts the level and lowercases it,
+so nginx's `notice`, Keycloak's `INFO` and LibreChat's `info` land on one scale
+— and the level selector has no `unknown` entry, because a line reaching the
+table always states one.
+
+**The scope selector actually scopes.** Each query carries both
+`job=~"${source:regex}"` and a literal job matcher. With a single selector,
+choosing "All" widened the query back to every stream in the cluster — which is
+how MongoDB and cert-manager ended up in an application view.
+
+Latency is populated for Kong only: the NCSA format the other sources use
+carries no timing field. That is a property of the format, not a gap in the
+query.
+
+### Parsed log tables### Parsed log tables
 
 Every dashboard shows its logs twice: a **table** with one column per parsed
 field, and, in a collapsed row underneath, the original **logs panel**.