Stéphane Tailland
26aa89643c
LibreChat served the lab CA's own certificate as its server certificate — a CA:TRUE certificate valid for the whole *.sttlab.local wildcard, from a hand-made tls.yaml that also carried the CA's private key. Gitignored, so never leaked, but the only service in the estate still copying the CA around instead of being issued a leaf. Now a cert-manager Certificate for librechat.sttlab.local: CA:FALSE, one dnsNames, key generated in the cluster and rotated there. The two uses of that Secret had to be separated first, which is the part worth knowing: sttlab-tls was both the ingress certificate and the CA bundle mounted as NODE_EXTRA_CA_CERTS for outbound calls. Replacing it with a leaf alone would have left LibreChat unable to verify Keycloak. The CA certificate now comes from ca-configmap.yaml — the certificate, never the key, committed like the agents'. Verified: the ingress serves CA:FALSE with SAN librechat.sttlab.local, the pod's outbound call to Keycloak returns 200 with TLS verified, and the old secret is deleted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
147 lines
4.4 KiB
Markdown
147 lines
4.4 KiB
Markdown
# LibreChat — Kubernetes Deployment
|
|
|
|
Deploys [LibreChat](https://www.librechat.ai/) on Kubernetes with MongoDB, MeiliSearch, and the JIRA MCP server.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Namespace: librechat
|
|
│
|
|
├── Ingress nginx (librechat.sttlab.pc — TLS wildcard *.sttlab.pc)
|
|
│ └── librechat :3080
|
|
│ ├── mongodb :27017 (StatefulSet, 5 Gi PVC)
|
|
│ └── meilisearch :7700 (Deployment, 2 Gi PVC)
|
|
│
|
|
└── MCP JIRA (namespace: mcp)
|
|
└── mcp-jira :9000 (streamable-http, 49 tools)
|
|
```
|
|
|
|
| Component | Image | Storage |
|
|
|-----------|-------|---------|
|
|
| LibreChat | `ghcr.io/danny-avila/librechat:latest` | — (stateless) |
|
|
| MongoDB | `mongo:7` | 5 Gi PVC |
|
|
| MeiliSearch | `getmeili/meilisearch:v1.7` | 2 Gi PVC |
|
|
|
|
## Directory structure
|
|
|
|
```
|
|
k8s/
|
|
├── namespace.yaml Namespace definition
|
|
├── secret.yaml Credentials (gitignored — never commit real values)
|
|
├── configmap.yaml Application environment config
|
|
├── kustomization.yaml Kustomize entry point
|
|
├── mongodb/
|
|
│ ├── pvc.yaml
|
|
│ ├── statefulset.yaml
|
|
│ └── service.yaml
|
|
├── meilisearch/
|
|
│ ├── pvc.yaml
|
|
│ ├── deployment.yaml
|
|
│ └── service.yaml
|
|
├── librechat/
|
|
│ ├── configmap-app.yaml Mounts librechat.yaml (MCP config, allowedDomains...)
|
|
│ ├── deployment.yaml
|
|
│ └── service.yaml
|
|
└── ingress/
|
|
├── ingress.yaml
|
|
├── certificate.yaml cert-manager Certificate → librechat-tls
|
|
└── (no TLS file: the key is generated in the cluster)
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
- Kubernetes cluster with an Nginx ingress controller
|
|
- A default StorageClass (for PVCs)
|
|
- `kubectl` configured against your cluster
|
|
- Wildcard TLS certificate for `*.sttlab.pc` in `$HOME/tls/`
|
|
|
|
## Initial deployment
|
|
|
|
### 1. Fill in secrets
|
|
|
|
Edit `secret.yaml` with real values. Generate them with:
|
|
|
|
```bash
|
|
openssl rand -base64 24 # MONGO_PASSWORD
|
|
openssl rand -base64 32 # MEILI_MASTER_KEY, JWT_SECRET, JWT_REFRESH_SECRET
|
|
openssl rand -hex 32 # CREDS_KEY
|
|
openssl rand -hex 8 # CREDS_IV
|
|
```
|
|
|
|
**Important:** `MONGO_URI` must include `?authSource=admin` because `MONGO_INITDB_ROOT_USERNAME` creates the user in the `admin` database:
|
|
```
|
|
mongodb://librechat:<password>@mongodb:27017/LibreChat?authSource=admin
|
|
```
|
|
|
|
### 2. TLS
|
|
|
|
Nothing to create. `ingress/certificate.yaml` asks cert-manager for a leaf
|
|
certificate for `librechat.sttlab.local`, issued by the `sttlab-local-ca`
|
|
ClusterIssuer into the `librechat-tls` Secret. The private key is generated in
|
|
the cluster, rotated automatically, and never exists as a file.
|
|
|
|
This replaced a hand-made `tls.yaml` holding the lab CA's own certificate *and
|
|
private key*, which LibreChat then served as its server certificate — a
|
|
`CA:TRUE` certificate valid for the whole `*.sttlab.local` wildcard where a leaf
|
|
for one name is what a server needs.
|
|
|
|
The CA **certificate** is still needed, for the outbound calls LibreChat makes
|
|
to Keycloak and the domain gateways. It lives in `ca-configmap.yaml`, is mounted
|
|
as `NODE_EXTRA_CA_CERTS`, and is committed — a certificate is public by
|
|
construction; the key is what must never be in a file.
|
|
|
|
### 3. Deploy
|
|
|
|
```bash
|
|
export DOMAIN=librechat.sttlab.pc
|
|
kubectl kustomize k8s/ | envsubst '${DOMAIN}' | kubectl apply -f -
|
|
```
|
|
|
|
> **Note:** `DOMAIN` must be exported before the pipe. An inline assignment (`DOMAIN=x cmd1 | cmd2`) does not propagate to `envsubst`.
|
|
|
|
### 4. Verify
|
|
|
|
```bash
|
|
kubectl get pods,ingress -n librechat
|
|
curl -sk https://librechat.sttlab.pc/health
|
|
```
|
|
|
|
## MCP JIRA
|
|
|
|
The JIRA MCP server runs in the `mcp` namespace (`mcp-jira:9000`, `streamable-http` transport).
|
|
|
|
`librechat/configmap-app.yaml` mounts `/app/librechat.yaml` into the container:
|
|
|
|
```yaml
|
|
mcpSettings:
|
|
allowedDomains:
|
|
- "http://mcp-jira.mcp.svc.cluster.local:9000"
|
|
mcpServers:
|
|
jira:
|
|
type: streamable-http
|
|
url: http://mcp-jira.mcp.svc.cluster.local:9000/mcp
|
|
```
|
|
|
|
> `mcpSettings.allowedDomains` is required to allow internal k8s domains, which are blocked by default as SSRF protection.
|
|
|
|
## Updating
|
|
|
|
```bash
|
|
export DOMAIN=librechat.sttlab.pc
|
|
kubectl kustomize k8s/ | envsubst '${DOMAIN}' | kubectl apply -f -
|
|
```
|
|
|
|
Force a pod restart:
|
|
|
|
```bash
|
|
kubectl rollout restart deployment/librechat -n librechat
|
|
```
|
|
|
|
## Teardown
|
|
|
|
```bash
|
|
kubectl delete namespace librechat
|
|
```
|
|
|
|
> This also deletes all PVCs and their data. Back up MongoDB first if needed.
|