fix: serve a leaf certificate, not the CA

LibreChat served the lab CA's own certificate as its server certificate — a
CA:TRUE certificate valid for the whole *.sttlab.local wildcard, from a
hand-made tls.yaml that also carried the CA's private key. Gitignored, so never
leaked, but the only service in the estate still copying the CA around instead
of being issued a leaf.

Now a cert-manager Certificate for librechat.sttlab.local: CA:FALSE, one
dnsNames, key generated in the cluster and rotated there.

The two uses of that Secret had to be separated first, which is the part worth
knowing: sttlab-tls was both the ingress certificate and the CA bundle mounted
as NODE_EXTRA_CA_CERTS for outbound calls. Replacing it with a leaf alone would
have left LibreChat unable to verify Keycloak. The CA certificate now comes from
ca-configmap.yaml — the certificate, never the key, committed like the agents'.

Verified: the ingress serves CA:FALSE with SAN librechat.sttlab.local, the pod's
outbound call to Keycloak returns 200 with TLS verified, and the old secret is
deleted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitignore

@@ -1,6 +1,8 @@
 # Secrets — ne jamais committer de valeurs réelles
 secret.yaml
-tls.yaml
+# tls.yaml : plus de fichier TLS local. Le certificat de service est émis par
+# cert-manager (ingress/certificate.yaml) et sa clé reste dans le cluster ; le
+# certificat du CA, lui, est public et versionné dans ca-configmap.yaml.
 
 # Certificats TLS
 *.crt

k8s/README.md

@@ -44,7 +44,8 @@ k8s/
 │   └── service.yaml
 └── ingress/
     ├── ingress.yaml
-    └── tls.yaml               TLS secret (gitignored — never commit)
+    ├── certificate.yaml       cert-manager Certificate → librechat-tls
+    └── (no TLS file: the key is generated in the cluster)
 ```
 
 ## Prerequisites
@@ -72,19 +73,22 @@ openssl rand -hex 8       # CREDS_IV
 mongodb://librechat:<password>@mongodb:27017/LibreChat?authSource=admin
 ```
 
-### 2. Create the TLS secret
+### 2. TLS
 
-```bash
+Nothing to create. `ingress/certificate.yaml` asks cert-manager for a leaf
-kubectl apply -f k8s/namespace.yaml
+certificate for `librechat.sttlab.local`, issued by the `sttlab-local-ca`
+ClusterIssuer into the `librechat-tls` Secret. The private key is generated in
+the cluster, rotated automatically, and never exists as a file.
 
-kubectl create secret tls sttlab-tls \
+This replaced a hand-made `tls.yaml` holding the lab CA's own certificate *and
-  --cert=$HOME/tls/sttlab.pc.crt \
+private key*, which LibreChat then served as its server certificate — a
-  --key=$HOME/tls/sttlab.pc.key \
+`CA:TRUE` certificate valid for the whole `*.sttlab.local` wildcard where a leaf
-  --namespace=librechat \
+for one name is what a server needs.
-  --dry-run=client -o yaml > k8s/ingress/tls.yaml
 
-kubectl apply -f k8s/ingress/tls.yaml
+The CA **certificate** is still needed, for the outbound calls LibreChat makes
-```
+to Keycloak and the domain gateways. It lives in `ca-configmap.yaml`, is mounted
+as `NODE_EXTRA_CA_CERTS`, and is committed — a certificate is public by
+construction; the key is what must never be in a file.
 
 ### 3. Deploy
 

k8s/ca-configmap.yaml

@@ -0,0 +1,38 @@
+# The lab CA, so LibreChat can verify keycloak.sttlab.local and the domain
+# gateways rather than disabling verification.
+#
+# The certificate, never the key — which is the whole difference from the
+# `ingress/tls.yaml` this replaces. A CA certificate is public by construction, so it is
+# committed and the deployment has no hidden prerequisite.
+#
+# Issued by the sttlab-local-ca ClusterIssuer, valid until 2026-11-24.
+# Regenerate on rotation:
+#   kubectl -n cert-manager get secret sttlab-local-ca \
+#     -o jsonpath='{.data.tls\.crt}' | base64 -d
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sttlab-ca
+  namespace: librechat
+data:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    MIIDPDCCAiSgAwIBAgIUd/utDf+f0yQR3v3qTjZqx+wvTZMwDQYJKoZIhvcNAQEL
+    BQAwGTEXMBUGA1UEAwwOKi5zdHRsYWIubG9jYWwwHhcNMjUxMTI0MTUzODQ1WhcN
+    MjYxMTI0MTUzODQ1WjAZMRcwFQYDVQQDDA4qLnN0dGxhYi5sb2NhbDCCASIwDQYJ
+    KoZIhvcNAQEBBQADggEPADCCAQoCggEBALj0sryq8qXtkShT/B+/EjVJNN/TLHIn
+    QJYc6xuUH/f7q3dYdLm4iH0XnFgrSQLUo4iqd/Mczef5iWCYwT5FFa7P17/TcLVf
+    gso0j41qNLBjGxDQGQ6Z1VCxi6oan8ow98rumjVj0E0Qgs5CX/rN+gcEmZ/TZFWU
+    qudVoztpeiVqJ0dxCIq+KMutzd1BU7BXx4VRO9Y+t1idUc7oY7BPmS2qPAN7TtXP
+    j1KhH2BgLsmF5GRhYHNS578eCltcmS2sbQ3c7GwOV96lpgF9TixEvFXeN4xk43E6
+    1COUD7CRiWAZ0XI8FMlpZGeNZeV4bbV/MqVig1ybWNOz9MDKYj3eoXUCAwEAAaN8
+    MHowHQYDVR0OBBYEFDbzYWOPr4vxa+VONs7xHbcNx70YMB8GA1UdIwQYMBaAFDbz
+    YWOPr4vxa+VONs7xHbcNx70YMA8GA1UdEwEB/wQFMAMBAf8wJwYDVR0RBCAwHoIO
+    Ki5zdHRsYWIubG9jYWyCDHN0dGxhYi5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAQEA
+    tU4p5/ONXm7x4YDzSeL8NdThVLguiFur0jcTxvvPUS4NFNsnQJk4OXxBgjy1YwpL
+    3i56/4mCEd9OCQF/Hl0eFwGZBzNKlMoTA08AC/lBDvPrnVjuUu0qe/b/2KOkhESa
+    x8rQNp0o8pcj0f29nu0JfRqYGkzy0Lh2qgXhPu2iC+t2PfdleRAky1tt68wogdFL
+    TvHDsJbVo/ihsPpONlSyHfxG6bb/0Mkz64PH46TTt9GvJflEKMqDwivNFUA1P5S/
+    U4Ic8Qapnxc7h2+WbRjmYrnuygC2wMBE6rL9OF3gwCr1fkgGGRr5lVRUq2X2HZV5
+    5jqViv6eurq8O61N3vWGCQ==
+    -----END CERTIFICATE-----

k8s/configmap.yaml

@@ -6,14 +6,29 @@ metadata:
 data:
   HOST: "0.0.0.0"
   PORT: "3080"
-  DOMAIN_CLIENT: "https://${DOMAIN}"
+  DOMAIN_CLIENT: "https://librechat.sttlab.local"
-  DOMAIN_SERVER: "https://${DOMAIN}"
+  DOMAIN_SERVER: "https://librechat.sttlab.local"
   MEILI_HOST: "http://meilisearch:7700"
   NODE_ENV: "production"
   ALLOW_EMAIL_LOGIN: "true"
   ALLOW_REGISTRATION: "true"
-  ALLOW_SOCIAL_LOGIN: "false"
+  ALLOW_SOCIAL_LOGIN: "true"
-  ALLOW_SOCIAL_REGISTRATION: "false"
+  ALLOW_SOCIAL_REGISTRATION: "true"
+  # Keycloak SSO, additive alongside email/password login (not replacing it
+  # — ALLOW_EMAIL_LOGIN stays true). OPENID_CLIENT_SECRET/OPENID_SESSION_SECRET
+  # are in librechat-secrets (see librechat/deployment.yaml).
+  OPENID_ISSUER: "https://keycloak.sttlab.local/realms/classic-models"
+  OPENID_CLIENT_ID: "librechat"
+  OPENID_CALLBACK_URL: "/oauth/openid/callback"
+  OPENID_SCOPE: "openid profile email librechat-audience"
+  OPENID_BUTTON_LABEL: "classic-models SSO"
+  # RP-initiated logout — without this, switching Keycloak users silently
+  # reuses the existing SSO session instead of re-prompting.
+  OPENID_USE_END_SESSION_ENDPOINT: "true"
   SESSION_EXPIRY: "900000"
   REFRESH_TOKEN_EXPIRY: "604800000"
   DEBUG_LOGGING: "false"
+  OTEL_TRACING_ENABLED: "true"
+  OTEL_SERVICE_NAME: "librechat"
+  OTEL_EXPORTER_OTLP_ENDPOINT: "http://tempo.observability.svc.cluster.local:4318"
+  OTEL_TRACES_EXPORTER: "otlp"

k8s/ingress/certificate.yaml

@@ -0,0 +1,21 @@
+# cert-manager issues this into the `librechat-tls` Secret.
+#
+# It replaces a hand-made `tls.yaml` that carried the lab CA's own certificate
+# and private key, which LibreChat then served as its server certificate — a
+# CA:TRUE certificate valid for the whole `*.sttlab.local` wildcard, where a
+# leaf for one name is what a server needs.
+#
+# The key is generated in the cluster, rotated by cert-manager, and never
+# reaches a file anyone has to remember not to commit.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: librechat-tls
+  namespace: librechat
+spec:
+  secretName: librechat-tls
+  dnsNames:
+    - librechat.sttlab.local
+  issuerRef:
+    name: sttlab-local-ca
+    kind: ClusterIssuer

k8s/ingress/ingress.yaml

@@ -9,17 +9,25 @@ metadata:
     nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
     nginx.ingress.kubernetes.io/proxy-buffering: "off"
     nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
-    nginx.ingress.kubernetes.io/configuration-snippet: |
+    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
-      proxy_set_header Upgrade $http_upgrade;
+    nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
-      proxy_set_header Connection "upgrade";
+    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "256k"
+    # configuration-snippet (WebSocket Upgrade/Connection headers) removed —
+    # this cluster's ingress-nginx has snippet annotations disabled
+    # (admission webhook rejects them outright). If any LibreChat feature
+    # turns out to need raw WebSocket upgrade specifically (vs. the SSE/
+    # HTTP-1.1 streaming already covered by proxy-buffering/proxy-http-version
+    # above), this needs a different mechanism — e.g. a ConfigMap-based
+    # global nginx snippet if the cluster admin allows one, since per-Ingress
+    # snippets are blocked here.
 spec:
   ingressClassName: nginx
   tls:
     - hosts:
-        - librechat.sttlab.pc
+        - librechat.sttlab.local
-      secretName: sttlab-tls
+      secretName: librechat-tls
   rules:
-    - host: librechat.sttlab.pc
+    - host: librechat.sttlab.local
       http:
         paths:
           - path: /

k8s/kustomization.yaml

@@ -5,6 +5,7 @@ resources:
   - namespace.yaml
   - secret.yaml
   - configmap.yaml
+  - ca-configmap.yaml
   - mongodb/pvc.yaml
   - mongodb/statefulset.yaml
   - mongodb/service.yaml
@@ -14,5 +15,5 @@ resources:
   - librechat/configmap-app.yaml
   - librechat/deployment.yaml
   - librechat/service.yaml
-  - ingress/tls.yaml
+  - ingress/certificate.yaml
   - ingress/ingress.yaml

k8s/librechat/configmap-app.yaml

@@ -5,11 +5,97 @@ metadata:
   namespace: librechat
 data:
   librechat.yaml: |
-    version: 1.3.9
+    version: 1.3.13
     mcpSettings:
       allowedDomains:
-        - "http://mcp-jira.mcp.svc.cluster.local:9000"
+        - "https://catalog-dev.sttlab.local"
+        - "https://customers-dev.sttlab.local"
+        - "https://payments-dev.sttlab.local"
+        - "https://sales-dev.sttlab.local"
+        - "https://workforce-dev.sttlab.local"
+        - "https://keycloak.sttlab.local"
     mcpServers:
-      jira:
+      catalog-mcp:
         type: streamable-http
-        url: http://mcp-jira.mcp.svc.cluster.local:9000/mcp
+        url: https://catalog-dev.sttlab.local/catalog-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/catalog-mcp/oauth/callback
+          scope: 'openid mcp-catalog-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      customers-mcp:
+        type: streamable-http
+        url: https://customers-dev.sttlab.local/customers-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/customers-mcp/oauth/callback
+          scope: 'openid mcp-customers-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      payments-mcp:
+        type: streamable-http
+        url: https://payments-dev.sttlab.local/payments-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/payments-mcp/oauth/callback
+          scope: 'openid mcp-payments-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      sales-mcp:
+        type: streamable-http
+        url: https://sales-dev.sttlab.local/sales-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/sales-mcp/oauth/callback
+          scope: 'openid mcp-sales-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+      workforce-mcp:
+        type: streamable-http
+        url: https://workforce-dev.sttlab.local/workforce-mcp
+        oauth:
+          authorization_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/auth
+          token_url: https://keycloak.sttlab.local/realms/classic-models/protocol/openid-connect/token
+          client_id: librechat
+          client_secret: '${LIBRECHAT_OIDC_CLIENT_SECRET}'
+          redirect_uri: https://librechat.sttlab.local/api/mcp/workforce-mcp/oauth/callback
+          scope: 'openid mcp-workforce-audience token-exchange-broker-audience'
+          grant_types_supported: ['authorization_code', 'refresh_token']
+          token_endpoint_auth_methods_supported: ['client_secret_post']
+    endpoints:
+      custom:
+        # kong-v2's platform-llm proxy (not the old kube/llms/openrouter lab)
+        # — key-auth, same shared-key limitation as before (LibreChat's
+        # custom endpoints don't support OAuth/OIDC on this outbound leg).
+        # baseURL is `/llm/v1`, not the full path: LibreChat appends
+        # `/chat/completions` itself.
+        - name: 'OpenRouter'
+          apiKey: '${PLATFORM_LLM_LIBRECHAT_APIKEY}'
+          baseURL: 'https://llm-dev.sttlab.local/llm/v1'
+          headers:
+            apikey: '${PLATFORM_LLM_LIBRECHAT_APIKEY}'
+          models:
+            default:
+              - 'google/gemini-3.7-flash'
+              - 'anthropic/claude-sonnet-5'
+              - 'openai/gpt-5.6-luna'
+              - 'deepseek/deepseek-v4-flash'
+              - 'openai/gpt-oss-120b'
+              - 'mistralai/mistral-nemo'
+            fetch: false
+          titleConvo: true
+          titleModel: 'google/gemini-3.7-flash'
+          modelDisplayLabel: 'OpenRouter'

k8s/librechat/deployment.yaml

@@ -12,7 +12,24 @@ spec:
     metadata:
       labels:
         app: librechat
+      annotations:
+        # Prometheus' dedicated `librechat` job (bearer token) targets this
+        # pod; the annotation is a selection marker, not a scrape config.
+        prometheus.io/scrape-with-auth: "true"
+        prometheus.io/port: "3080"
     spec:
+      # *.sttlab.local are lab-only hostnames (self-signed cert), not real
+      # DNS — this pod needs an explicit mapping to each Gateway's stable
+      # in-cluster ClusterIP to resolve them at all.
+      # catalog-dev: kong-v2 catalog-api Gateway (dataplane-ingress-catalog-dev-v4bkm-xqq6v), REST+MCP share this hostname now.
+      # llm-dev: kong-v2's platform-llm Gateway (dataplane-ingress-llm-dev-rlrxf-7fkb7).
+      hostAliases:
+        - ip: "192.168.194.225"
+          hostnames:
+            - "catalog-dev.sttlab.local"
+        - ip: "192.168.194.153"
+          hostnames:
+            - "llm-dev.sttlab.local"
       initContainers:
         - name: wait-for-mongodb
           image: busybox:1.36
@@ -81,13 +98,58 @@ spec:
                 secretKeyRef:
                   name: librechat-secrets
                   key: ANTHROPIC_API_KEY
+            - name: PLATFORM_LLM_LIBRECHAT_APIKEY
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: PLATFORM_LLM_LIBRECHAT_APIKEY
+            - name: LIBRECHAT_OIDC_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: LIBRECHAT_OIDC_CLIENT_SECRET
+            # Same Keycloak client secret as LIBRECHAT_OIDC_CLIENT_SECRET
+            # above (one Keycloak client, "librechat", used both for the
+            # app's own SSO login and for each MCP server's OAuth flow) —
+            # LibreChat's OIDC-login code path reads this specific env var
+            # name, so it's duplicated rather than referenced once.
+            - name: OPENID_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: LIBRECHAT_OIDC_CLIENT_SECRET
+            - name: OPENID_SESSION_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: OPENID_SESSION_SECRET
+            # Enables /metrics (401 without it). Scraped by the Prometheus in
+            # the observability namespace using a Bearer token.
+            - name: METRICS_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: librechat-secrets
+                  key: METRICS_SECRET
+            # jira-mcp.sttlab.local / konggw.sttlab.local serve a self-signed
+            # cert (see hostAliases comment above) — Node rejects it by
+            # default; NODE_EXTRA_CA_CERTS trusts just this one CA, unlike
+            # NODE_TLS_REJECT_UNAUTHORIZED=0 which would disable TLS
+            # verification globally.
+            - name: NODE_EXTRA_CA_CERTS
+              value: /etc/sttlab-ca/ca.crt
+            # 512Mi was too tight for LibreChat under real use (MCP clients,
+            # OTel instrumentation) and caused repeated V8 heap-limit crashes.
+            # Cap old-space explicitly, leaving headroom under the new
+            # container limit for non-heap memory (buffers, native modules).
+            - name: NODE_OPTIONS
+              value: "--max-old-space-size=1200"
           resources:
             requests:
               cpu: 100m
-              memory: 256Mi
+              memory: 512Mi
             limits:
               cpu: 500m
-              memory: 512Mi
+              memory: 1536Mi
           livenessProbe:
             httpGet:
               path: /health
@@ -104,7 +166,13 @@ spec:
             - name: app-config
               mountPath: /app/librechat.yaml
               subPath: librechat.yaml
+            - name: sttlab-ca
+              mountPath: /etc/sttlab-ca
+              readOnly: true
       volumes:
         - name: app-config
           configMap:
             name: librechat-app-config
+        - name: sttlab-ca
+          configMap:
+            name: sttlab-ca

k8s/secret.example.yaml

@@ -15,3 +15,16 @@ stringData:
   # Optional: fill in your AI provider API keys
   OPENAI_API_KEY: ""
   ANTHROPIC_API_KEY: ""
+  # platform-llm proxy consumer key (kong-v2/platform-llm, consumer "librechat")
+  PLATFORM_LLM_LIBRECHAT_APIKEY: ""
+  LIBRECHAT_OIDC_CLIENT_SECRET: ""
+  OPENID_SESSION_SECRET: ""
+
+  # Enables the built-in Prometheus endpoint (/metrics on port 3080).
+  # Without it LibreChat answers 401. Prometheus scrapes it with
+  # `Authorization: Bearer <METRICS_SECRET>`: the SAME value must exist in the
+  # `librechat-metrics` secret of the observability namespace
+  # (see infra-observability/k8s/prometheus/). A Kubernetes Secret is
+  # namespaced, so the value has to be duplicated — one side verifies the
+  # token, the other presents it.
+  METRICS_SECRET: "CHANGE_ME_METRICS_SECRET_MIN_32_CHARS"