first commit

2026-05-31 12:18:37 +00:00
parent 4f1c251ff8
commit e525b053ec
18 changed files with 974 additions and 104 deletions
@@ -0,0 +1,338 @@
+# Gravitee APIM — k3s single-node deployment
+
+Production-like deployment of Gravitee APIM OSS on k3s with end-to-end TLS, credentials injected from Kubernetes secrets, and an internal CA managed by cert-manager.
+
+---
+
+## Architecture
+
+```
+Internet
+  └── nginx ingress (TLS termination + HTTPS re-encryption to backends)
+        ├── console.gravitee.sttlab.pc  →  graviteeio-apim-ui (nginx, port 8080)
+        ├── portal.gravitee.sttlab.pc   →  graviteeio-apim-portal (nginx, port 8080)
+        ├── api.gravitee.sttlab.pc      →  graviteeio-apim-api (Jetty/HTTPS, port 8083)
+        └── gateway.gravitee.sttlab.pc  →  graviteeio-apim-gateway (Vert.x/HTTPS, port 8082)
+
+Internal backends:
+  graviteeio-apim-api     ──TLS──► mongodb:27017
+  graviteeio-apim-api     ──TLS──► gravitee-es-master:9200
+  graviteeio-apim-gateway ──TLS──► mongodb:27017
+  graviteeio-apim-gateway ──TLS──► gravitee-es-master:9200
+```
+
+**Stack:**
+- **Gravitee APIM 4.x** — Management API (Jetty), Gateway (Vert.x/Netty), Console UI and Developer Portal (Angular/nginx)
+- **Gravitee Kubernetes Operator (GKO)** — GitOps management of APIs via CRDs
+- **MongoDB 8.x** (Bitnami) — management and rate limiting persistence
+- **Elasticsearch 8.x** (Elastic) — analytics and reporting
+- **cert-manager** — internal self-signed PKI with automatic renewal
+- **nginx ingress** — external TLS termination + TLS proxy to backends
+
+---
+
+## PKI and certificates
+
+### Trust chain
+
+```
+gravitee-selfsigned-bootstrap (Issuer, self-signed)
+  └── gravitee-ca (Certificate isCA:true) → secret: gravitee-ca-tls
+        └── gravitee-ca-issuer (CA Issuer)
+              ├── console-tls          (ingress console)
+              ├── portal-tls           (ingress portal)
+              ├── api-tls              (ingress api)
+              ├── gateway-tls          (ingress gateway)
+              ├── mongodb-tls          (MongoDB server)
+              ├── elasticsearch-tls    (ES server + truststore.jks)
+              ├── api-internal-tls     (Jetty API server + keystore.p12)
+              └── gateway-internal-tls (Vert.x gateway server + keystore.p12)
+```
+
+### cert-manager keystores
+
+Three secrets include PKCS12 keystores automatically generated by cert-manager:
+- `api-internal-tls` → `keystore.p12` (Jetty server cert) + `truststore.p12` (CA trust for JVM)
+- `gateway-internal-tls` → `keystore.p12` (Vert.x server cert) + `truststore.p12` (CA trust for JVM)
+- `elasticsearch-tls` → `keystore.p12` + `truststore.p12` (ES server cert — keystore unused by Gravitee)
+
+Each component mounts only its own internal TLS secret at `/run/secrets/tls/`, providing both
+the server keystore and the JVM truststore from a single volume.
+
+Keystore password is stored in the `gravitee-jks-password` secret.
+
+### Notable SANs
+
+- `api-internal-tls` includes `api.gravitee.sttlab.pc` — required for nginx → pod TLS verification
+- `gateway-internal-tls` includes `gateway.gravitee.sttlab.pc` — same reason
+- `elasticsearch-tls` includes `gravitee-es-master.gravitee-apim.svc.cluster.local`
+
+### Browser trust
+
+Export the CA for browser import:
+```bash
+kubectl get secret gravitee-ca-tls -n gravitee-apim \
+  -o jsonpath='{.data.ca\.crt}' | base64 -d > gravitee-ca.crt
+```
+Import `gravitee-ca.crt` into the OS keychain or browser trust store.
+
+---
+
+## Kubernetes secrets
+
+All secrets are created by `secrets-create.sh` before the first deployment.
+
+| Secret | Contents | Consumed by |
+|--------|----------|-------------|
+| `mongodb-credentials` | root/gravitee MongoDB passwords | Bitnami MongoDB chart |
+| `gravitee-mongodb-uri` | `GRAVITEE_MANAGEMENT_MONGODB_URI`, `GRAVITEE_RATELIMIT_MONGODB_URI` | api, gateway (Secret Manager) |
+| `gravitee-jwt` | `GRAVITEE_JWT_SECRET` | api (Secret Manager) |
+| `gravitee-admin` | `admin-password-plain`, `admin-password-bcrypt` | api (Secret Manager) |
+| `gravitee-es-master-credentials` | ES `username`, `password` | api, gateway (Secret Manager) |
+| `gravitee-encryption` | `api-properties-encryption-secret` | api, gateway (Secret Manager) |
+| `gravitee-jks-password` | keystore password | api, gateway (`JAVA_OPTS` env + Secret Manager for `ssl.keystore.password`) + cert-manager |
+| `gravitee-ca-trust` | `ca.crt` only — no tls.crt/key | nginx ingress `proxy-ssl-secret` |
+
+> `gravitee-ca-trust` is created by `secrets-create.sh` after the certificates are ready (it reads `ca.crt` from `gravitee-ca-tls`). It must contain **only** `ca.crt` — if `tls.crt`/`tls.key` were present, nginx would present them as a client certificate, triggering unintended mTLS toward backends.
+
+### Credential injection
+
+Credentials are resolved at runtime by the **Gravitee Secret Manager** (Kubernetes provider, enabled via `secrets.kubernetes`). Helm values reference secrets using the `secret://kubernetes/<secret-name>:<key>` syntax, which is rendered verbatim into `gravitee.yml` and resolved by Gravitee on startup.
+
+```yaml
+# In apim-values.yml — resolved by Gravitee Secret Manager at runtime
+jwtSecret: "secret://kubernetes/gravitee-jwt:GRAVITEE_JWT_SECRET"
+mongo:
+  uri: "secret://kubernetes/gravitee-mongodb-uri:GRAVITEE_MANAGEMENT_MONGODB_URI"
+es:
+  security:
+    username: "secret://kubernetes/gravitee-es-master-credentials:username"
+    password: "secret://kubernetes/gravitee-es-master-credentials:password"
+```
+
+The only secret still injected as a Kubernetes env var is `gravitee-jks-password` (`JKS_PASSWORD`), because it is needed in `JAVA_OPTS` before the JVM starts — before the Gravitee Secret Manager is available.
+
+---
+
+## Internal TLS
+
+### MongoDB
+
+TLS enabled via `tls=true` in the connection URI. Server certificate validation is handled by the JVM truststore:
+```
+JAVA_OPTS: -Djavax.net.ssl.trustStoreType=PKCS12
+           -Djavax.net.ssl.trustStore=/run/secrets/tls/truststore.p12
+           -Djavax.net.ssl.trustStorePassword=${JKS_PASSWORD}
+```
+The `truststore.p12` is sourced from the component's own internal TLS secret (`api-internal-tls` or `gateway-internal-tls`), which cert-manager populates with the issuing CA chain.
+
+MongoDB runs with `--tlsAllowConnectionsWithoutCertificates` (no client mTLS) but requires server TLS (`--tlsMode=requireTLS`).
+
+### Elasticsearch
+
+HTTPS with basic auth. Server certificate signed by the Gravitee CA, validated by the same JVM truststore. The HTTPS endpoint and credentials are configured in `apim-values.yml` via `es.endpoints` / `es.security.*` and resolved at runtime by the Gravitee Secret Manager.
+
+### nginx → backends (proxy TLS)
+
+nginx verifies backend certificates:
+```yaml
+nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
+nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
+nginx.ingress.kubernetes.io/configuration-snippet: |
+  proxy_ssl_name api.gravitee.sttlab.pc;
+```
+
+- `proxy-ssl-secret` provides the CA for chain verification
+- `proxy_ssl_name` (via snippet) tells nginx which hostname to check against the backend cert SANs
+- `proxy_ssl_server_name on` is intentionally **absent**: it modifies the TLS ClientHello in a way that triggers an `illegal parameter` alert in Jetty 12
+
+Snippets require `annotations-risk-level: Critical` in the `ingress-nginx-controller` ConfigMap.
+
+---
+
+## UI ingress (SPA)
+
+The console and portal are Angular SPAs. The Gravitee chart injects `rewrite-target: /$1` by default. Without a capture group in the path, all asset requests are rewritten to `/` (index.html served instead of CSS/JS).
+
+Fix applied:
+```yaml
+path: /(.*)
+pathType: ImplementationSpecific
+annotations:
+  nginx.ingress.kubernetes.io/use-regex: "true"
+  nginx.ingress.kubernetes.io/rewrite-target: /$1
+```
+
+---
+
+## Deployment
+
+### Prerequisites
+
+- k3s with nginx ingress and cert-manager installed
+- Helm repos added: `bitnami`, `elastic`, `graviteeio`
+- `htpasswd` available (for admin BCrypt hash generation)
+
+### First install
+
+```bash
+./deploy.sh
+```
+
+Or step by step:
+
+```bash
+# 1. Create credential secrets
+./secrets-create.sh
+
+# 2. Deploy PKI and wait for certificates
+kubectl apply -f certificates.yml
+kubectl -n gravitee-apim wait --for=condition=Ready certificate --all --timeout=180s
+
+# 3. Create CA trust secret for nginx (requires certificates to be ready first)
+kubectl -n gravitee-apim get secret gravitee-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | \
+  kubectl -n gravitee-apim create secret generic gravitee-ca-trust --from-file=ca.crt=/dev/stdin \
+  --dry-run=client -o yaml | kubectl apply -f -
+
+# 4. Enable nginx ingress snippets
+kubectl patch configmap ingress-nginx-controller -n ingress-nginx \
+  --type merge -p '{"data":{"allow-snippet-annotations":"true","annotations-risk-level":"Critical"}}'
+kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx
+
+# 5. Deploy backends
+helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
+helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
+
+# 6. Deploy Gravitee
+helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
+
+# 7. Install GKO (after APIM is ready)
+# Create GKO credentials secret first (see GKO section below)
+helm upgrade --install gko graviteeio/gko \
+  --namespace gravitee-apim \
+  --set manager.httpClient.trustStore.path=/etc/ssl/certs/gravitee-ca.crt \
+  --set-json 'manager.volumes=[{"name":"gravitee-ca","secret":{"secretName":"gravitee-ca-tls","items":[{"key":"ca.crt","path":"gravitee-ca.crt"}]}}]' \
+  --set-json 'manager.volumeMounts=[{"name":"gravitee-ca","mountPath":"/etc/ssl/certs/gravitee-ca.crt","subPath":"gravitee-ca.crt","readOnly":true}]'
+
+# 8. Apply GKO resources
+kubectl apply -f gko-management-context.yaml
+kubectl apply -f gko-api.yaml
+```
+
+### Rebuild workloads only (secrets and certs preserved)
+
+```bash
+helm uninstall graviteeio-apim mongodb elasticsearch -n gravitee-apim
+kubectl delete pvc -n gravitee-apim --all
+
+helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
+helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
+helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
+```
+
+> Secrets, certs, PVs and the `gravitee-ca-trust` secret are preserved across workload rebuilds.
+
+### /etc/hosts
+
+```
+192.168.1.18  console.gravitee.sttlab.pc portal.gravitee.sttlab.pc api.gravitee.sttlab.pc gateway.gravitee.sttlab.pc
+```
+
+---
+
+## Access
+
+| URL | Description |
+|-----|-------------|
+| https://console.gravitee.sttlab.pc | Management console |
+| https://portal.gravitee.sttlab.pc | Developer portal |
+| https://api.gravitee.sttlab.pc/management | Management API |
+| https://gateway.gravitee.sttlab.pc | Gateway (data plane) |
+
+**Admin credentials:**
+```bash
+# Username: admin
+# Password (plaintext):
+kubectl get secret gravitee-admin -n gravitee-apim \
+  -o jsonpath='{.data.admin-password-plain}' | base64 -d && echo
+```
+
+---
+
+## Gravitee Kubernetes Operator (GKO)
+
+GKO enables GitOps-driven API management: APIs, plans and subscriptions are declared as Kubernetes CRDs and automatically synced to the Management API.
+
+### Installation
+
+```bash
+helm upgrade --install gko graviteeio/gko \
+  --namespace gravitee-apim \
+  --set manager.httpClient.trustStore.path=/etc/ssl/certs/gravitee-ca.crt \
+  --set-json 'manager.volumes=[{"name":"gravitee-ca","secret":{"secretName":"gravitee-ca-tls","items":[{"key":"ca.crt","path":"gravitee-ca.crt"}]}}]' \
+  --set-json 'manager.volumeMounts=[{"name":"gravitee-ca","mountPath":"/etc/ssl/certs/gravitee-ca.crt","subPath":"gravitee-ca.crt","readOnly":true}]'
+```
+
+The Gravitee CA is mounted into the GKO pod so its HTTP client trusts the Management API's certificate (port 83, HTTPS). Without this, GKO would fail to connect when validating or reconciling CRDs.
+
+### GKO secrets
+
+| Secret | Contents | Purpose |
+|--------|----------|---------|
+| `gko-management-credentials` | `username`, `password` | GKO → Management API auth via `ManagementContext.spec.auth.secretRef` |
+
+Create after deploying APIM:
+```bash
+ADMIN_PLAIN=$(kubectl get secret gravitee-admin -n gravitee-apim \
+  -o jsonpath='{.data.admin-password-plain}' | base64 -d)
+kubectl create secret generic gko-management-credentials -n gravitee-apim \
+  --from-literal=username=admin \
+  --from-literal=password="${ADMIN_PLAIN}"
+```
+
+### CRDs
+
+| CRD | Description |
+|-----|-------------|
+| `ManagementContext` | Connection to a Gravitee Management API instance |
+| `ApiV4Definition` | Gravitee v4 API (proxy, message, native) |
+| `ApiDefinition` | Gravitee v2 API (legacy) |
+| `Application` | Consumer application |
+| `ApiResource` | Shared resource (cache, OAuth2, etc.) |
+| `SharedPolicyGroup` | Reusable policy group |
+| `Subscription` | API subscription |
+
+### ManagementContext
+
+`gko-management-context.yaml` — connects GKO to the local Management API:
+```yaml
+spec:
+  baseUrl: https://graviteeio-apim-api.gravitee-apim.svc.cluster.local:83
+  environmentId: DEFAULT
+  organizationId: DEFAULT
+  auth:
+    secretRef:
+      name: gko-management-credentials
+      namespace: gravitee-apim
+```
+
+> Port 83 is the Kubernetes service port (maps to container port 8083).
+
+### Deploying an API
+
+```bash
+kubectl apply -f gko-management-context.yaml
+kubectl apply -f gko-api.yaml
+kubectl get apiv4definition -n gravitee-apim
+```
+
+The `processingStatus: Completed` and `state: STARTED` in the status confirm the API is live on the gateway.
+
+---
+
+## Security notes
+
+- The Gravitee CA is self-signed. Replace with a Vault PKI Issuer in production (see comment in `certificates.yml`).
+- `proxy-ssl-verify: "on"` is enabled on all backend ingresses — nginx validates pod certificates.
+- Client mTLS is not enabled for MongoDB (complexity vs. benefit for intra-cluster traffic).
+- `annotations-risk-level: Critical` on the ingress controller is required for `proxy_ssl_name` snippets — document this in the ops runbook.