Send traces to Tempo and expose metrics to Prometheus

Point the OTLP exporter at Tempo instead of Jaeger.

Enable the built-in /metrics endpoint by setting METRICS_SECRET: without
it LibreChat answers 401 and exposes nothing. Prometheus scrapes it with a
Bearer token from a dedicated job, since annotation-based discovery cannot
carry credentials.

A Kubernetes Secret is namespaced, so the same value must also exist as
`librechat-metrics` in the observability namespace — one side verifies the
token, the other presents it. secret.example.yaml documents the pairing;
the real secret.yaml stays gitignored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

k8s/configmap.yaml

@@ -30,5 +30,5 @@ data:
   DEBUG_LOGGING: "false"
   OTEL_TRACING_ENABLED: "true"
   OTEL_SERVICE_NAME: "librechat"
-  OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger.observability.svc.cluster.local:4318"
+  OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo.observability.svc.cluster.local:4318"
   OTEL_TRACES_EXPORTER: "otlp"

k8s/librechat/deployment.yaml

@@ -12,6 +12,11 @@ spec:
     metadata:
       labels:
         app: librechat
+      annotations:
+        # Prometheus' dedicated `librechat` job (bearer token) targets this
+        # pod; the annotation is a selection marker, not a scrape config.
+        prometheus.io/scrape-with-auth: "true"
+        prometheus.io/port: "3080"
     spec:
       # *.sttlab.local are lab-only hostnames (self-signed cert), not real
       # DNS — this pod needs an explicit mapping to each Gateway's stable
@@ -118,6 +123,13 @@ spec:
                 secretKeyRef:
                   name: librechat-secrets
                   key: OPENID_SESSION_SECRET
+            # Enables /metrics (401 without it). Scraped by the Prometheus in
+            # the observability namespace using a Bearer token.
+            - name: METRICS_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: METRICS_SECRET
             # jira-mcp.sttlab.local / konggw.sttlab.local serve a self-signed
             # cert (see hostAliases comment above) — Node rejects it by
             # default; NODE_EXTRA_CA_CERTS trusts just this one CA, unlike

k8s/secret.example.yaml

@@ -19,3 +19,12 @@ stringData:
   PLATFORM_LLM_LIBRECHAT_APIKEY: ""
   LIBRECHAT_OIDC_CLIENT_SECRET: ""
   OPENID_SESSION_SECRET: ""
+
+  # Enables the built-in Prometheus endpoint (/metrics on port 3080).
+  # Without it LibreChat answers 401. Prometheus scrapes it with
+  # `Authorization: Bearer <METRICS_SECRET>`: the SAME value must exist in the
+  # `librechat-metrics` secret of the observability namespace
+  # (see infra-observability/k8s/prometheus/). A Kubernetes Secret is
+  # namespaced, so the value has to be duplicated — one side verifies the
+  # token, the other presents it.
+  METRICS_SECRET: "CHANGE_ME_METRICS_SECRET_MIN_32_CHARS"