fix: serve a leaf certificate, not the CA

LibreChat served the lab CA's own certificate as its server certificate — a CA:TRUE certificate valid for the whole *.sttlab.local wildcard, from a hand-made tls.yaml that also carried the CA's private key. Gitignored, so never leaked, but the only service in the estate still copying the CA around instead of being issued a leaf. Now a cert-manager Certificate for librechat.sttlab.local: CA:FALSE, one dnsNames, key generated in the cluster and rotated there. The two uses of that Secret had to be separated first, which is the part worth knowing: sttlab-tls was both the ingress certificate and the CA bundle mounted as NODE_EXTRA_CA_CERTS for outbound calls. Replacing it with a leaf alone would have left LibreChat unable to verify Keycloak. The CA certificate now comes from ca-configmap.yaml — the certificate, never the key, committed like the agents'. Verified: the ingress serves CA:FALSE with SAN librechat.sttlab.local, the pod's outbound call to Keycloak returns 200 with TLS verified, and the old secret is deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
refactor: request the neutrally-named token-exchange broker
2026-08-29 18:01:00 +02:00 · 2026-08-29 07:53:00 +02:00 · 2026-08-27 18:18:36 +02:00 · 2026-08-27 17:29:17 +02:00 · 2026-08-25 07:27:14 +02:00 · 2026-08-23 17:22:08 +02:00
10 changed files with 285 additions and 29 deletions
@@ -1,6 +1,8 @@
 # Secrets — ne jamais committer de valeurs réelles
 secret.yaml
-tls.yaml
+# tls.yaml : plus de fichier TLS local. Le certificat de service est émis par
+# cert-manager (ingress/certificate.yaml) et sa clé reste dans le cluster ; le
+# certificat du CA, lui, est public et versionné dans ca-configmap.yaml.

 # Certificats TLS
 *.crt
@@ -44,7 +44,8 @@ k8s/
 │   └── service.yaml
 └── ingress/
    ├── ingress.yaml
-    └── tls.yaml               TLS secret (gitignored — never commit)
+    ├── certificate.yaml       cert-manager Certificate → librechat-tls
+    └── (no TLS file: the key is generated in the cluster)
 ```

 ## Prerequisites
@@ -72,19 +73,22 @@ openssl rand -hex 8       # CREDS_IV
 mongodb://librechat:<password>@mongodb:27017/LibreChat?authSource=admin
 ```

-### 2. Create the TLS secret
+### 2. TLS

-```bash
-kubectl apply -f k8s/namespace.yaml
+Nothing to create. `ingress/certificate.yaml` asks cert-manager for a leaf
+certificate for `librechat.sttlab.local`, issued by the `sttlab-local-ca`
+ClusterIssuer into the `librechat-tls` Secret. The private key is generated in
+the cluster, rotated automatically, and never exists as a file.

-kubectl create secret tls sttlab-tls \
-  --cert=$HOME/tls/sttlab.pc.crt \
-  --key=$HOME/tls/sttlab.pc.key \
-  --namespace=librechat \
-  --dry-run=client -o yaml > k8s/ingress/tls.yaml
+This replaced a hand-made `tls.yaml` holding the lab CA's own certificate *and
+private key*, which LibreChat then served as its server certificate — a
+`CA:TRUE` certificate valid for the whole `*.sttlab.local` wildcard where a leaf
+for one name is what a server needs.

-kubectl apply -f k8s/ingress/tls.yaml
-```
+The CA **certificate** is still needed, for the outbound calls LibreChat makes
+to Keycloak and the domain gateways. It lives in `ca-configmap.yaml`, is mounted
+as `NODE_EXTRA_CA_CERTS`, and is committed — a certificate is public by
+construction; the key is what must never be in a file.

 ### 3. Deploy

@@ -0,0 +1,38 @@
+# The lab CA, so LibreChat can verify keycloak.sttlab.local and the domain
+# gateways rather than disabling verification.
+#
+# The certificate, never the key — which is the whole difference from the
+# `ingress/tls.yaml` this replaces. A CA certificate is public by construction, so it is
+# committed and the deployment has no hidden prerequisite.
+#
+# Issued by the sttlab-local-ca ClusterIssuer, valid until 2026-11-24.
+# Regenerate on rotation:
+#   kubectl -n cert-manager get secret sttlab-local-ca \
+#     -o jsonpath='{.data.tls\.crt}' | base64 -d
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sttlab-ca
+  namespace: librechat
+data:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    MIIDPDCCAiSgAwIBAgIUd/utDf+f0yQR3v3qTjZqx+wvTZMwDQYJKoZIhvcNAQEL
+    BQAwGTEXMBUGA1UEAwwOKi5zdHRsYWIubG9jYWwwHhcNMjUxMTI0MTUzODQ1WhcN
+    MjYxMTI0MTUzODQ1WjAZMRcwFQYDVQQDDA4qLnN0dGxhYi5sb2NhbDCCASIwDQYJ
+    KoZIhvcNAQEBBQADggEPADCCAQoCggEBALj0sryq8qXtkShT/B+/EjVJNN/TLHIn
+    QJYc6xuUH/f7q3dYdLm4iH0XnFgrSQLUo4iqd/Mczef5iWCYwT5FFa7P17/TcLVf
+    gso0j41qNLBjGxDQGQ6Z1VCxi6oan8ow98rumjVj0E0Qgs5CX/rN+gcEmZ/TZFWU
+    qudVoztpeiVqJ0dxCIq+KMutzd1BU7BXx4VRO9Y+t1idUc7oY7BPmS2qPAN7TtXP
+    j1KhH2BgLsmF5GRhYHNS578eCltcmS2sbQ3c7GwOV96lpgF9TixEvFXeN4xk43E6
+    1COUD7CRiWAZ0XI8FMlpZGeNZeV4bbV/MqVig1ybWNOz9MDKYj3eoXUCAwEAAaN8
+    MHowHQYDVR0OBBYEFDbzYWOPr4vxa+VONs7xHbcNx70YMB8GA1UdIwQYMBaAFDbz
+    YWOPr4vxa+VONs7xHbcNx70YMA8GA1UdEwEB/wQFMAMBAf8wJwYDVR0RBCAwHoIO
+    Ki5zdHRsYWIubG9jYWyCDHN0dGxhYi5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAQEA
+    tU4p5/ONXm7x4YDzSeL8NdThVLguiFur0jcTxvvPUS4NFNsnQJk4OXxBgjy1YwpL
+    3i56/4mCEd9OCQF/Hl0eFwGZBzNKlMoTA08AC/lBDvPrnVjuUu0qe/b/2KOkhESa
+    x8rQNp0o8pcj0f29nu0JfRqYGkzy0Lh2qgXhPu2iC+t2PfdleRAky1tt68wogdFL
+    TvHDsJbVo/ihsPpONlSyHfxG6bb/0Mkz64PH46TTt9GvJflEKMqDwivNFUA1P5S/
+    U4Ic8Qapnxc7h2+WbRjmYrnuygC2wMBE6rL9OF3gwCr1fkgGGRr5lVRUq2X2HZV5
+    5jqViv6eurq8O61N3vWGCQ==
+    -----END CERTIFICATE-----
@@ -6,14 +6,29 @@ metadata:
 data:
  HOST: "0.0.0.0"
  PORT: "3080"
-  DOMAIN_CLIENT: "https://${DOMAIN}"
-  DOMAIN_SERVER: "https://${DOMAIN}"
+  DOMAIN_CLIENT: "https://librechat.sttlab.local"
+  DOMAIN_SERVER: "https://librechat.sttlab.local"
  MEILI_HOST: "http://meilisearch:7700"
  NODE_ENV: "production"
  ALLOW_EMAIL_LOGIN: "true"
  ALLOW_REGISTRATION: "true"
-  ALLOW_SOCIAL_LOGIN: "false"
-  ALLOW_SOCIAL_REGISTRATION: "false"
+  ALLOW_SOCIAL_LOGIN: "true"
+  ALLOW_SOCIAL_REGISTRATION: "true"
+  # Keycloak SSO, additive alongside email/password login (not replacing it
+  # — ALLOW_EMAIL_LOGIN stays true). OPENID_CLIENT_SECRET/OPENID_SESSION_SECRET
+  # are in librechat-secrets (see librechat/deployment.yaml).
+  OPENID_ISSUER: "https://keycloak.sttlab.local/realms/classic-models"
+  OPENID_CLIENT_ID: "librechat"
+  OPENID_CALLBACK_URL: "/oauth/openid/callback"
+  OPENID_SCOPE: "openid profile email librechat-audience"
+  OPENID_BUTTON_LABEL: "classic-models SSO"
+  # RP-initiated logout — without this, switching Keycloak users silently
+  # reuses the existing SSO session instead of re-prompting.
+  OPENID_USE_END_SESSION_ENDPOINT: "true"
  SESSION_EXPIRY: "900000"
  REFRESH_TOKEN_EXPIRY: "604800000"
  DEBUG_LOGGING: "false"
+  OTEL_TRACING_ENABLED: "true"
+  OTEL_SERVICE_NAME: "librechat"
+  OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo.observability.svc.cluster.local:4318"
+  OTEL_TRACES_EXPORTER: "otlp"
@@ -0,0 +1,21 @@
+# cert-manager issues this into the `librechat-tls` Secret.
+#
+# It replaces a hand-made `tls.yaml` that carried the lab CA's own certificate
+# and private key, which LibreChat then served as its server certificate — a
+# CA:TRUE certificate valid for the whole `*.sttlab.local` wildcard, where a
+# leaf for one name is what a server needs.
+#
+# The key is generated in the cluster, rotated by cert-manager, and never
+# reaches a file anyone has to remember not to commit.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: librechat-tls
+  namespace: librechat
+spec:
+  secretName: librechat-tls
+  dnsNames:
+    - librechat.sttlab.local
+  issuerRef:
+    name: sttlab-local-ca
+    kind: ClusterIssuer
@@ -9,17 +9,25 @@ metadata:
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
-    nginx.ingress.kubernetes.io/configuration-snippet: |
-      proxy_set_header Upgrade $http_upgrade;
-      proxy_set_header Connection "upgrade";
+    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
+    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
+    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "256k"
+    # configuration-snippet (WebSocket Upgrade/Connection headers) removed —
+    # this cluster's ingress-nginx has snippet annotations disabled
+    # (admission webhook rejects them outright). If any LibreChat feature
+    # turns out to need raw WebSocket upgrade specifically (vs. the SSE/
+    # HTTP-1.1 streaming already covered by proxy-buffering/proxy-http-version
+    # above), this needs a different mechanism — e.g. a ConfigMap-based
+    # global nginx snippet if the cluster admin allows one, since per-Ingress
+    # snippets are blocked here.
 spec:
  ingressClassName: nginx
  tls:
    - hosts:
-        - librechat.sttlab.pc
-      secretName: sttlab-tls
+        - librechat.sttlab.local
+      secretName: librechat-tls
  rules:
-    - host: librechat.sttlab.pc
+    - host: librechat.sttlab.local
      http:
        paths:
          - path: /
@@ -5,6 +5,7 @@ resources:
  - namespace.yaml
  - secret.yaml
  - configmap.yaml
+  - ca-configmap.yaml
  - mongodb/pvc.yaml
  - mongodb/statefulset.yaml
  - mongodb/service.yaml
@@ -14,5 +15,5 @@ resources:
  - librechat/configmap-app.yaml
  - librechat/deployment.yaml
  - librechat/service.yaml
-  - ingress/tls.yaml
+  - ingress/certificate.yaml
  - ingress/ingress.yaml
@@ -5,11 +5,97 @@ metadata:
  namespace: librechat
 data:
  librechat.yaml: |
-    version: 1.3.9
+    version: 1.3.13
    mcpSettings:
      allowedDomains:
-        - "http://mcp-jira.mcp.svc.cluster.local:9000"
+        - "https://catalog-dev.sttlab.local"
+        - "https://customers-dev.sttlab.local"
+        - "https://payments-dev.sttlab.local"
+        - "https://sales-dev.sttlab.local"
+        - "https://workforce-dev.sttlab.local"
+        - "https://keycloak.sttlab.local"
    mcpServers:
-      jira:
+      catalog-mcp:
        type: streamable-http
-        url: http://mcp-jira.mcp.svc.cluster.local:9000/mcp
+        url: https://catalog-dev.sttlab.local/catalog-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/catalog-mcp/oauth/callback
+          scope: 'openid mcp-catalog-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      customers-mcp:
+        type: streamable-http
+        url: https://customers-dev.sttlab.local/customers-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/customers-mcp/oauth/callback
+          scope: 'openid mcp-customers-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      payments-mcp:
+        type: streamable-http
+        url: https://payments-dev.sttlab.local/payments-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/payments-mcp/oauth/callback
+          scope: 'openid mcp-payments-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      sales-mcp:
+        type: streamable-http
+        url: https://sales-dev.sttlab.local/sales-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/sales-mcp/oauth/callback
+          scope: 'openid mcp-sales-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      workforce-mcp:
+        type: streamable-http
+        url: https://workforce-dev.sttlab.local/workforce-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/workforce-mcp/oauth/callback
+          scope: 'openid mcp-workforce-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+    endpoints:
+      custom:
+        # kong-v2's platform-llm proxy (not the old kube/llms/openrouter lab)
+        # — key-auth, same shared-key limitation as before (LibreChat's
+        # custom endpoints don't support OAuth/OIDC on this outbound leg).
+        # baseURL is `/llm/v1`, not the full path: LibreChat appends
+        # `/chat/completions` itself.
+        - name: 'OpenRouter'
+          apiKey: '${PLATFORM_LLM_LIBRECHAT_APIKEY}'
+          baseURL: 'https://llm-dev.sttlab.local/llm/v1'
+          headers:
+            apikey: '${PLATFORM_LLM_LIBRECHAT_APIKEY}'
+          models:
+            default:
+              - 'google/gemini-3.7-flash'
+              - 'anthropic/claude-sonnet-5'
+              - 'openai/gpt-5.6-luna'
+              - 'deepseek/deepseek-v4-flash'
+              - 'openai/gpt-oss-120b'
+              - 'mistralai/mistral-nemo'
+            fetch: false
+          titleConvo: true
+          titleModel: 'google/gemini-3.7-flash'
+          modelDisplayLabel: 'OpenRouter'
@@ -12,7 +12,24 @@ spec:
    metadata:
      labels:
        app: librechat
+      annotations:
+        # Prometheus' dedicated `librechat` job (bearer token) targets this
+        # pod; the annotation is a selection marker, not a scrape config.
+        prometheus.io/scrape-with-auth: "true"
+        prometheus.io/port: "3080"
    spec:
+      # *.sttlab.local are lab-only hostnames (self-signed cert), not real
+      # DNS — this pod needs an explicit mapping to each Gateway's stable
+      # in-cluster ClusterIP to resolve them at all.
+      # catalog-dev: kong-v2 catalog-api Gateway (dataplane-ingress-catalog-dev-v4bkm-xqq6v), REST+MCP share this hostname now.
+      # llm-dev: kong-v2's platform-llm Gateway (dataplane-ingress-llm-dev-rlrxf-7fkb7).
+      hostAliases:
+        - ip: "192.168.194.225"
+          hostnames:
+            - "catalog-dev.sttlab.local"
+        - ip: "192.168.194.153"
+          hostnames:
+            - "llm-dev.sttlab.local"
      initContainers:
        - name: wait-for-mongodb
          image: busybox:1.36
@@ -81,13 +98,58 @@ spec:
                secretKeyRef:
                  name: librechat-secrets
                  key: ANTHROPIC_API_KEY
+            - name: PLATFORM_LLM_LIBRECHAT_APIKEY
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: PLATFORM_LLM_LIBRECHAT_APIKEY
+            - name: LIBRECHAT_OIDC_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: LIBRECHAT_OIDC_CLIENT_SECRET
+            # Same Keycloak client secret as LIBRECHAT_OIDC_CLIENT_SECRET
+            # above (one Keycloak client, "librechat", used both for the
+            # app's own SSO login and for each MCP server's OAuth flow) —
+            # LibreChat's OIDC-login code path reads this specific env var
+            # name, so it's duplicated rather than referenced once.
+            - name: OPENID_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: LIBRECHAT_OIDC_CLIENT_SECRET
+            - name: OPENID_SESSION_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: OPENID_SESSION_SECRET
+            # Enables /metrics (401 without it). Scraped by the Prometheus in
+            # the observability namespace using a Bearer token.
+            - name: METRICS_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: METRICS_SECRET
+            # jira-mcp.sttlab.local / konggw.sttlab.local serve a self-signed
+            # cert (see hostAliases comment above) — Node rejects it by
+            # default; NODE_EXTRA_CA_CERTS trusts just this one CA, unlike
+            # NODE_TLS_REJECT_UNAUTHORIZED=0 which would disable TLS
+            # verification globally.
+            - name: NODE_EXTRA_CA_CERTS
+              value: /etc/sttlab-ca/ca.crt
+            # 512Mi was too tight for LibreChat under real use (MCP clients,
+            # OTel instrumentation) and caused repeated V8 heap-limit crashes.
+            # Cap old-space explicitly, leaving headroom under the new
+            # container limit for non-heap memory (buffers, native modules).
+            - name: NODE_OPTIONS
+              value: "--max-old-space-size=1200"
          resources:
            requests:
              cpu: 100m
-              memory: 256Mi
+              memory: 512Mi
            limits:
              cpu: 500m
-              memory: 512Mi
+              memory: 1536Mi
          livenessProbe:
            httpGet:
              path: /health
@@ -104,7 +166,13 @@ spec:
            - name: app-config
              mountPath: /app/librechat.yaml
              subPath: librechat.yaml
+            - name: sttlab-ca
+              mountPath: /etc/sttlab-ca
+              readOnly: true
      volumes:
        - name: app-config
          configMap:
            name: librechat-app-config
+        - name: sttlab-ca
+          configMap:
+            name: sttlab-ca
@@ -15,3 +15,16 @@ stringData:
  # Optional: fill in your AI provider API keys
  OPENAI_API_KEY: ""
  ANTHROPIC_API_KEY: ""
+  # platform-llm proxy consumer key (kong-v2/platform-llm, consumer "librechat")
+  PLATFORM_LLM_LIBRECHAT_APIKEY: ""
+  LIBRECHAT_OIDC_CLIENT_SECRET: ""
+  OPENID_SESSION_SECRET: ""
+
+  # Enables the built-in Prometheus endpoint (/metrics on port 3080).
+  # Without it LibreChat answers 401. Prometheus scrapes it with
+  # `Authorization: Bearer <METRICS_SECRET>`: the SAME value must exist in the
+  # `librechat-metrics` secret of the observability namespace
+  # (see infra-observability/k8s/prometheus/). A Kubernetes Secret is
+  # namespaced, so the value has to be duplicated — one side verifies the
+  # token, the other presents it.
+  METRICS_SECRET: "CHANGE_ME_METRICS_SECRET_MIN_32_CHARS"
Author	SHA1	Message	Date
Stéphane Tailland	26aa89643c	fix: serve a leaf certificate, not the CA LibreChat served the lab CA's own certificate as its server certificate — a CA:TRUE certificate valid for the whole *.sttlab.local wildcard, from a hand-made tls.yaml that also carried the CA's private key. Gitignored, so never leaked, but the only service in the estate still copying the CA around instead of being issued a leaf. Now a cert-manager Certificate for librechat.sttlab.local: CA:FALSE, one dnsNames, key generated in the cluster and rotated there. The two uses of that Secret had to be separated first, which is the part worth knowing: sttlab-tls was both the ingress certificate and the CA bundle mounted as NODE_EXTRA_CA_CERTS for outbound calls. Replacing it with a leaf alone would have left LibreChat unable to verify Keycloak. The CA certificate now comes from ca-configmap.yaml — the certificate, never the key, committed like the agents'. Verified: the ingress serves CA:FALSE with SAN librechat.sttlab.local, the pod's outbound call to Keycloak returns 200 with TLS verified, and the old secret is deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 18:01:00 +02:00
Stéphane Tailland	bfc748a925	refactor: request the neutrally-named token-exchange broker Follows the rename in the realm. The client that performs the on-behalf-of exchange is no longer named after the middleware that calls it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-29 07:53:00 +02:00
Stéphane Tailland	7892f0f8f5	Remove the Grafana MCP server Reverts the declaration. The approach does not hold up in practice: an MCP client resends the whole tool list every turn, so simply having this server connected cost ~19k prompt tokens per request — and ~5.4k even after trimming the tool set by 72%. On top of that its metric-name listing returns 3330 entries, 137 KiB, which the assistant called twice while exploring. Conversations tripped Kong's 50k-tokens-per-minute AI rate limit and sat in a 429 retry loop. The deployment, the Vault-backed secret and the Grafana service account stay in place; only the LibreChat wiring goes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 18:18:36 +02:00
Stéphane Tailland	2a695da0dd	Declare the Grafana MCP server Prometheus, Loki and Tempo become tools the assistant can call, so the telemetry can be questioned rather than browsed — which is where the trace dashboard kept running out of road. Verified: LibreChat initialises the server in 168ms and loads its tool list. No oauth block, unlike every other server here. The others sit behind a Kong gateway enforcing openid-connect; this one is reached directly on the cluster network and authenticates to Grafana with a service account token of its own. LibreChat passes no user identity along, so there is no per-user scoping: every LibreChat user gets the same full read of all telemetry, including the Kong audit records and the LLM prompts now captured there. That is stated in the config rather than left to be discovered, and it is the reason to move this server behind a gateway like the others. allowedDomains gains the first http:// and first cluster-internal entry; without it LibreChat's SSRF protection refuses the address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-27 17:29:17 +02:00
Stéphane Tailland	6aad64903b	Send traces to Tempo and expose metrics to Prometheus Point the OTLP exporter at Tempo instead of Jaeger. Enable the built-in /metrics endpoint by setting METRICS_SECRET: without it LibreChat answers 401 and exposes nothing. Prometheus scrapes it with a Bearer token from a dedicated job, since annotation-based discovery cannot carry credentials. A Kubernetes Secret is namespaced, so the same value must also exist as `librechat-metrics` in the observability namespace — one side verifies the token, the other presents it. secret.example.yaml documents the pairing; the real secret.yaml stays gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-25 07:27:14 +02:00
Stéphane Tailland	40c6e9d723	Update of MCP servers	2026-08-23 17:22:08 +02:00
Stéphane Tailland	2836f42996	fix(mcp): Synchronize allowedDomains in configmap-app	2026-08-23 15:54:16 +02:00
Stéphane Tailland	8c363cf463	fix(ingress): Add proxy buffer annotations for SSE streams	2026-08-23 08:12:21 +02:00
Stéphane Tailland	ffe5b1cb12	fix(mcp): Add keycloak.sttlab.local to mcpSettings.allowedDomains	2026-08-23 08:02:51 +02:00
Stéphane Tailland	e235d3dea4	fix(oidc): Switch LibreChat and MCP OAuth to local Keycloak (keycloak.sttlab.local)	2026-08-23 07:55:28 +02:00