feat: deploy n8n on the cluster

The runtime for the estate's autonomous agents. Postgres as a StatefulSet with
its own claim, n8n behind a cert-manager certificate and the nginx ingress, and
both secrets in Vault behind an AppRole scoped to n8n/* alone.

The certificate is issued into the cluster rather than copied into the repo, so
the private key never reaches git. The CA certificate is committed and mounted
as NODE_EXTRA_CA_CERTS, so a workflow can reach keycloak and the domain
gateways without anyone disabling verification to make it work.

One replica and strategy: Recreate, both deliberate — n8n needs Redis and queue
mode before a second scheduler is safe, and a ReadWriteOnce volume deadlocks a
rolling update.

N8N_ENCRYPTION_KEY is generated once and kept in Vault. Regenerating it does not
re-encrypt the stored credentials, it orphans them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -0,0 +1,79 @@
+# infra-n8n
+
+n8n on the lab cluster: the runtime for the estate's autonomous agents,
+`Approvisionnement` and `Recouvrement`, whose workflows live in
+`classic-models-agent-flows`.
+
+Nothing business-specific is here. This repo deploys the engine; the workflows
+are imported into it.
+
+## Layout
+
+| | |
+|---|---|
+| `k8s/postgres/` | the database, a StatefulSet with its own volume claim |
+| `k8s/n8n/` | the workload, its config and its data volume |
+| `k8s/ingress/` | a cert-manager `Certificate` and the nginx `Ingress` |
+| `k8s/secrets/` | the `ClusterSecretStore` for this estate and the `ExternalSecret` |
+| `dev/` | the Flux plumbing that applies `k8s/`, and manages itself |
+
+Reachable at **https://n8n.sttlab.local** once `192.168.139.101` (ingress-nginx)
+resolves for that name.
+
+## Two secrets, both of which must outlive any pod
+
+They live in Vault at `secrets/n8n/instance`, read through the `vault-n8n`
+store — an AppRole whose policy can read `n8n/*` and nothing else, so a
+compromise here is not a way into the keycloak estate's secrets.
+
+- `DB_PASSWORD`
+- `N8N_ENCRYPTION_KEY` — encrypts the credentials n8n stores in its database.
+  **Never regenerate it.** A new key does not re-encrypt what is stored; it
+  orphans it. The workflows survive, every credential in them does not.
+
+## Bootstrap
+
+Three manual applies, once. The Vault side is set up first — the commands are in
+`k8s/secrets/clustersecretstore.yaml`.
+
+```
+kubectl apply -f k8s/namespace.yaml
+kubectl apply -f dev/gitrepository.yaml
+kubectl apply -f dev/bootstrap-kustomization.yaml
+```
+
+After that everything reconciles from git, `dev/` included. Never `kubectl
+apply` the objects under `k8s/` afterwards — Flux reverts them within its
+interval and the cluster silently stops matching what you edited.
+
+## Choices worth knowing
+
+**One replica, and `strategy: Recreate`.** n8n splits editor from workers only
+in queue mode, which needs Redis; without it a second replica is two schedulers
+racing over the same cron triggers. And the data volume is ReadWriteOnce, so a
+rolling update would deadlock — the new pod waiting for a volume the old one has
+not released.
+
+**The certificate is issued, not committed.** cert-manager writes `n8n-tls` from
+the `sttlab-local-ca` ClusterIssuer. The private key stays in the cluster.
+
+**The CA certificate *is* committed**, in `k8s/ca-configmap.yaml`, and mounted
+as `NODE_EXTRA_CA_CERTS`. It is a certificate, never a key, and checking it in
+means a workflow can call `keycloak.sttlab.local` and the domain gateways
+without anyone disabling verification to make it work. Regenerate on rotation —
+it expires 2026-11-24.
+
+**`N8N_HOST` and `WEBHOOK_URL` are the ingress host, not the Service.** They go
+into the webhook URLs n8n hands out; advertised as the in-cluster name, they
+would be URLs nothing outside can call.
+
+**Executions are pruned after two weeks.** The default keeps them for ever,
+which turns the database into a log.
+
+## What is deliberately not here
+
+Authentication. n8n's own user management is left at its defaults, and the
+instance is reachable to anyone who can resolve the host on the lab network.
+Before it holds anything that matters, it needs either n8n's SSO against the
+`classic-models` realm or an `openid-connect` filter at the ingress — the same
+Keycloak the rest of the estate already uses.

dev/bootstrap-kustomization.yaml

@@ -0,0 +1,30 @@
+# Watches ./dev, so the GitRepository and the Kustomization below it reconcile
+# from git like any other manifest. It lives in ./dev itself, so it also manages
+# itself — the arrangement `flux bootstrap` sets up, which flux-operator does not
+# create, and the same one the rest of the estate uses.
+#
+# Three manual applies, once, to close the loop — nothing can watch ./dev until
+# the namespace and the source that serves it exist:
+#
+#   kubectl apply -f k8s/namespace.yaml
+#   kubectl apply -f dev/gitrepository.yaml
+#   kubectl apply -f dev/bootstrap-kustomization.yaml
+#
+# After that every file in this repo, this one included, reconciles from git.
+#
+# Sharp edge: with prune enabled, deleting flux-kustomization.yaml from git
+# deletes the Kustomization it defines, and that Kustomization's own prune then
+# removes the deployment — including, on the PVCs, the workflows. The
+# volumeClaimTemplate on Postgres survives that; n8n-data does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: infra-n8n-bootstrap
+  namespace: n8n
+spec:
+  interval: 10m
+  sourceRef:
+    kind: GitRepository
+    name: infra-n8n
+  path: ./dev
+  prune: true

dev/flux-kustomization.yaml

@@ -0,0 +1,14 @@
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: infra-n8n
+  namespace: n8n
+spec:
+  interval: 10m
+  sourceRef:
+    kind: GitRepository
+    name: infra-n8n
+  path: ./k8s
+  prune: true
+  wait: true
+  timeout: 5m

dev/gitrepository.yaml

@@ -0,0 +1,14 @@
+# HTTPS with no credentials: the repo is anonymously readable over HTTPS on
+# gitea, so a deploy key would guard nothing while adding a Vault entry and a
+# gitea key to create by hand before anything deploys. TLS still gives integrity
+# and authenticity of the source.
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: infra-n8n
+  namespace: n8n
+spec:
+  interval: 5m
+  url: https://gitea.sttlab.eu/stt/infra-n8n.git
+  ref:
+    branch: main

k8s/ca-configmap.yaml

@@ -0,0 +1,36 @@
+# The lab CA, so n8n can verify keycloak.sttlab.local and the domain gateways
+# rather than disabling verification. A CA certificate is public by
+# construction — this is the certificate, never the key, and it is checked in so
+# the deployment has no hidden prerequisite.
+#
+# Issued by the sttlab-local-ca ClusterIssuer, valid until 2026-11-24.
+# Regenerate on rotation:
+#   kubectl -n cert-manager get secret sttlab-local-ca \
+#     -o jsonpath='{.data.tls\.crt}' | base64 -d
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sttlab-ca
+  namespace: n8n
+data:
+  ca.crt: |
+    -----BEGIN CERTIFICATE-----
+    MIIDPDCCAiSgAwIBAgIUd/utDf+f0yQR3v3qTjZqx+wvTZMwDQYJKoZIhvcNAQEL
+    BQAwGTEXMBUGA1UEAwwOKi5zdHRsYWIubG9jYWwwHhcNMjUxMTI0MTUzODQ1WhcN
+    MjYxMTI0MTUzODQ1WjAZMRcwFQYDVQQDDA4qLnN0dGxhYi5sb2NhbDCCASIwDQYJ
+    KoZIhvcNAQEBBQADggEPADCCAQoCggEBALj0sryq8qXtkShT/B+/EjVJNN/TLHIn
+    QJYc6xuUH/f7q3dYdLm4iH0XnFgrSQLUo4iqd/Mczef5iWCYwT5FFa7P17/TcLVf
+    gso0j41qNLBjGxDQGQ6Z1VCxi6oan8ow98rumjVj0E0Qgs5CX/rN+gcEmZ/TZFWU
+    qudVoztpeiVqJ0dxCIq+KMutzd1BU7BXx4VRO9Y+t1idUc7oY7BPmS2qPAN7TtXP
+    j1KhH2BgLsmF5GRhYHNS578eCltcmS2sbQ3c7GwOV96lpgF9TixEvFXeN4xk43E6
+    1COUD7CRiWAZ0XI8FMlpZGeNZeV4bbV/MqVig1ybWNOz9MDKYj3eoXUCAwEAAaN8
+    MHowHQYDVR0OBBYEFDbzYWOPr4vxa+VONs7xHbcNx70YMB8GA1UdIwQYMBaAFDbz
+    YWOPr4vxa+VONs7xHbcNx70YMA8GA1UdEwEB/wQFMAMBAf8wJwYDVR0RBCAwHoIO
+    Ki5zdHRsYWIubG9jYWyCDHN0dGxhYi5sb2NhbDANBgkqhkiG9w0BAQsFAAOCAQEA
+    tU4p5/ONXm7x4YDzSeL8NdThVLguiFur0jcTxvvPUS4NFNsnQJk4OXxBgjy1YwpL
+    3i56/4mCEd9OCQF/Hl0eFwGZBzNKlMoTA08AC/lBDvPrnVjuUu0qe/b/2KOkhESa
+    x8rQNp0o8pcj0f29nu0JfRqYGkzy0Lh2qgXhPu2iC+t2PfdleRAky1tt68wogdFL
+    TvHDsJbVo/ihsPpONlSyHfxG6bb/0Mkz64PH46TTt9GvJflEKMqDwivNFUA1P5S/
+    U4Ic8Qapnxc7h2+WbRjmYrnuygC2wMBE6rL9OF3gwCr1fkgGGRr5lVRUq2X2HZV5
+    5jqViv6eurq8O61N3vWGCQ==
+    -----END CERTIFICATE-----

k8s/ingress/certificate.yaml

@@ -0,0 +1,17 @@
+# cert-manager issues this into the `n8n-tls` Secret, from the same
+# ClusterIssuer as the rest of the estate.
+#
+# Deliberately not a copy of a certificate committed to git: the private key
+# stays in the cluster, is rotated by cert-manager, and never reaches a repo.
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: n8n-tls
+  namespace: n8n
+spec:
+  secretName: n8n-tls
+  dnsNames:
+    - n8n.sttlab.local
+  issuerRef:
+    name: sttlab-local-ca
+    kind: ClusterIssuer

k8s/ingress/ingress.yaml

@@ -0,0 +1,31 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: n8n
+  namespace: n8n
+  annotations:
+    # The editor pushes execution progress over a long-lived connection, and
+    # imports can be large. Buffering it defeats the first; the default body
+    # size defeats the second.
+    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
+    nginx.ingress.kubernetes.io/proxy-buffering: "off"
+    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - n8n.sttlab.local
+      secretName: n8n-tls
+  rules:
+    - host: n8n.sttlab.local
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: n8n
+                port:
+                  number: 80

k8s/kustomization.yaml

@@ -0,0 +1,15 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - ca-configmap.yaml
+  - secrets/clustersecretstore.yaml
+  - secrets/externalsecret.yaml
+  - postgres/statefulset.yaml
+  - postgres/service.yaml
+  - n8n/pvc.yaml
+  - n8n/configmap.yaml
+  - n8n/deployment.yaml
+  - n8n/service.yaml
+  - ingress/certificate.yaml
+  - ingress/ingress.yaml

k8s/n8n/configmap.yaml

@@ -0,0 +1,35 @@
+# Nothing here is sensitive. The two values that are — the database password and
+# the credential encryption key — arrive through n8n-secrets.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: n8n-config
+  namespace: n8n
+data:
+  GENERIC_TIMEZONE: Europe/Paris
+  TZ: Europe/Paris
+  NODE_ENV: production
+
+  # What n8n believes it is reachable at. It goes into the webhook URLs it
+  # generates and into the links it puts in its own UI, so it must be the
+  # ingress host and not the Service — a webhook advertised as `n8n` inside the
+  # cluster is a webhook nothing outside can call.
+  N8N_HOST: n8n.sttlab.local
+  N8N_PORT: "5678"
+  N8N_PROTOCOL: https
+  WEBHOOK_URL: https://n8n.sttlab.local/
+  N8N_EDITOR_BASE_URL: https://n8n.sttlab.local/
+
+  # TLS terminates at the ingress, so n8n itself serves plain HTTP and must be
+  # told it is behind a proxy — otherwise it reads the hop's address instead of
+  # the caller's.
+  N8N_PROXY_HOPS: "1"
+
+  # The default is to keep every execution for ever, which turns the database
+  # into a log. Two weeks is long enough to debug a failed run.
+  EXECUTIONS_DATA_PRUNE: "true"
+  EXECUTIONS_DATA_MAX_AGE: "336"
+
+  N8N_DIAGNOSTICS_ENABLED: "false"
+  N8N_VERSION_NOTIFICATIONS_ENABLED: "false"
+  N8N_RUNNERS_ENABLED: "true"

k8s/n8n/deployment.yaml

@@ -0,0 +1,110 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: n8n
+  namespace: n8n
+spec:
+  # One replica. n8n's queue mode splits the editor from workers and needs Redis
+  # to do it; without that, a second replica is two schedulers racing over the
+  # same cron triggers. See the README before scaling this.
+  replicas: 1
+  strategy:
+    # The volume is ReadWriteOnce, so a rolling update deadlocks: the new pod
+    # waits for a volume the old pod has not released.
+    type: Recreate
+  selector:
+    matchLabels:
+      app: n8n
+  template:
+    metadata:
+      labels:
+        app: n8n
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        fsGroup: 1000
+      initContainers:
+        - name: wait-for-postgres
+          image: postgres:16.10-alpine
+          command:
+            - sh
+            - -c
+            - |
+              until pg_isready -h n8n-postgres -U n8n -d n8n; do
+                echo "waiting for postgres..."; sleep 3
+              done
+      containers:
+        - name: n8n
+          image: docker.n8n.io/n8nio/n8n:1.128.2
+          ports:
+            - name: http
+              containerPort: 5678
+          env:
+            - name: DB_TYPE
+              value: postgresdb
+            - name: DB_POSTGRESDB_HOST
+              value: n8n-postgres
+            - name: DB_POSTGRESDB_PORT
+              value: "5432"
+            - name: DB_POSTGRESDB_DATABASE
+              value: n8n
+            - name: DB_POSTGRESDB_USER
+              value: n8n
+            - name: DB_POSTGRESDB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: n8n-secrets
+                  key: DB_PASSWORD
+            # Encrypts stored credentials. Held in Vault, never regenerated —
+            # a new key does not re-encrypt, it orphans everything already
+            # stored.
+            - name: N8N_ENCRYPTION_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: n8n-secrets
+                  key: N8N_ENCRYPTION_KEY
+            # The lab CA, so a workflow can call keycloak.sttlab.local and the
+            # domain gateways without disabling verification. n8n is Node, hence
+            # this variable rather than a trust store.
+            - name: NODE_EXTRA_CA_CERTS
+              value: /etc/sttlab-ca/ca.crt
+          envFrom:
+            - configMapRef:
+                name: n8n-config
+          volumeMounts:
+            - name: data
+              mountPath: /home/node/.n8n
+            - name: sttlab-ca
+              mountPath: /etc/sttlab-ca
+              readOnly: true
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 15
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: http
+            initialDelaySeconds: 60
+            periodSeconds: 20
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: "2"
+              memory: 2Gi
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+      volumes:
+        - name: data
+          persistentVolumeClaim:
+            claimName: n8n-data
+        - name: sttlab-ca
+          configMap:
+            name: sttlab-ca

k8s/n8n/pvc.yaml

@@ -0,0 +1,13 @@
+# n8n keeps its own state here — including the SQLite of an unconfigured
+# install, which is not used since the database is Postgres, but also binary
+# data and anything a node writes. Small, and it must survive a restart.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: n8n-data
+  namespace: n8n
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 2Gi

k8s/n8n/service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: n8n
+  namespace: n8n
+spec:
+  selector:
+    app: n8n
+  ports:
+    - name: http
+      port: 80
+      targetPort: http

k8s/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: n8n

k8s/postgres/service.yaml

@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: n8n-postgres
+  namespace: n8n
+spec:
+  selector:
+    app: n8n-postgres
+  ports:
+    - name: postgres
+      port: 5432
+      targetPort: postgres

k8s/postgres/statefulset.yaml

@@ -0,0 +1,71 @@
+# A StatefulSet rather than a Deployment: one identity, one volume, and no
+# window where two pods hold the same data directory. The volumeClaimTemplate
+# also means the claim is not deleted with the workload — losing the workflows
+# to a `kubectl delete deploy` would be an avoidable way to lose them.
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: n8n-postgres
+  namespace: n8n
+spec:
+  serviceName: n8n-postgres
+  replicas: 1
+  selector:
+    matchLabels:
+      app: n8n-postgres
+  template:
+    metadata:
+      labels:
+        app: n8n-postgres
+    spec:
+      securityContext:
+        fsGroup: 999
+      containers:
+        - name: postgres
+          image: postgres:16.10-alpine
+          ports:
+            - name: postgres
+              containerPort: 5432
+          env:
+            - name: POSTGRES_USER
+              value: n8n
+            - name: POSTGRES_DB
+              value: n8n
+            - name: POSTGRES_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: n8n-secrets
+                  key: DB_PASSWORD
+            # The image initialises into this directory. Naming a subdirectory
+            # of the mount rather than the mount itself keeps initdb away from
+            # the lost+found some volume types leave at the root.
+            - name: PGDATA
+              value: /var/lib/postgresql/data/pgdata
+          volumeMounts:
+            - name: data
+              mountPath: /var/lib/postgresql/data
+          readinessProbe:
+            exec:
+              command: ["pg_isready", "-U", "n8n", "-d", "n8n"]
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            exec:
+              command: ["pg_isready", "-U", "n8n", "-d", "n8n"]
+            initialDelaySeconds: 30
+            periodSeconds: 20
+          resources:
+            requests:
+              cpu: 50m
+              memory: 256Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+  volumeClaimTemplates:
+    - metadata:
+        name: data
+      spec:
+        accessModes: ["ReadWriteOnce"]
+        resources:
+          requests:
+            storage: 5Gi

k8s/secrets/clustersecretstore.yaml

@@ -0,0 +1,37 @@
+# One AppRole per estate, each with a policy that can read only its own path.
+# The n8n store cannot read `keycloak/*` and the keycloak store cannot read
+# `n8n/*` — so a compromised workload in one namespace does not become a way
+# into the other's secrets.
+#
+# Its Vault side is created by hand, once:
+#
+#   vault policy write n8n-readonly -  <<'HCL'
+#   path "secrets/data/n8n/*"     { capabilities = ["read"] }
+#   path "secrets/metadata/n8n/*" { capabilities = ["list", "read"] }
+#   HCL
+#   vault write auth/approle/role/n8n-eso token_policies=n8n-readonly \
+#     token_ttl=1h token_max_ttl=4h
+#   kubectl -n external-secrets create secret generic vault-approle-n8n \
+#     --from-literal=roleId=$(vault read -field=role_id auth/approle/role/n8n-eso/role-id) \
+#     --from-literal=secretId=$(vault write -f -field=secret_id auth/approle/role/n8n-eso/secret-id)
+apiVersion: external-secrets.io/v1
+kind: ClusterSecretStore
+metadata:
+  name: vault-n8n
+spec:
+  provider:
+    vault:
+      server: https://vault.sttlab.eu
+      path: secrets
+      version: v2
+      auth:
+        appRole:
+          path: approle
+          roleRef:
+            name: vault-approle-n8n
+            namespace: external-secrets
+            key: roleId
+          secretRef:
+            name: vault-approle-n8n
+            namespace: external-secrets
+            key: secretId

k8s/secrets/externalsecret.yaml

@@ -0,0 +1,22 @@
+# Two values, both of which must outlive any pod.
+#
+# N8N_ENCRYPTION_KEY encrypts the credentials n8n stores in its database. Lose
+# it and every stored credential becomes unreadable — the workflows survive, the
+# credentials do not. It is generated once and kept in Vault for that reason,
+# never regenerated on a redeploy.
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: n8n-secrets
+  namespace: n8n
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    kind: ClusterSecretStore
+    name: vault-n8n
+  target:
+    name: n8n-secrets
+    creationPolicy: Owner
+  dataFrom:
+    - extract:
+        key: n8n/instance