stt/infra-gravitee-apim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
sttlab
2026-05-31 12:18:37 +00:00
parent 4f1c251ff8
commit e525b053ec
18 changed files with 974 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
-258
View File
@@ -1,258 +0,0 @@
# Gravitee APIM — k3s single-node deployment
Production-like deployment of Gravitee APIM OSS on k3s with end-to-end TLS, credentials injected from Kubernetes secrets, and an internal CA managed by cert-manager.
---
## Architecture
```
Internet
└── nginx ingress (TLS termination + HTTPS re-encryption to backends)
├── console.gravitee.sttlab.pc → graviteeio-apim-ui (nginx, port 8080)
├── portal.gravitee.sttlab.pc → graviteeio-apim-portal (nginx, port 8080)
├── api.gravitee.sttlab.pc → graviteeio-apim-api (Jetty/HTTPS, port 8083)
└── gateway.gravitee.sttlab.pc → graviteeio-apim-gateway (Vert.x/HTTPS, port 8082)
Internal backends:
graviteeio-apim-api ──TLS──► mongodb:27017
graviteeio-apim-api ──TLS──► gravitee-es-master:9200
graviteeio-apim-gateway ──TLS──► mongodb:27017
graviteeio-apim-gateway ──TLS──► gravitee-es-master:9200
```
**Stack:**
- **Gravitee APIM 4.x** — Management API (Jetty), Gateway (Vert.x/Netty), Console UI and Developer Portal (Angular/nginx)
- **MongoDB 8.x** (Bitnami) — management and rate limiting persistence
- **Elasticsearch 8.x** (Elastic) — analytics and reporting
- **cert-manager** — internal self-signed PKI with automatic renewal
- **nginx ingress** — external TLS termination + TLS proxy to backends
---
## PKI and certificates
### Trust chain
```
gravitee-selfsigned-bootstrap (Issuer, self-signed)
└── gravitee-ca (Certificate isCA:true) → secret: gravitee-ca-tls
└── gravitee-ca-issuer (CA Issuer)
├── console-tls (ingress console)
├── portal-tls (ingress portal)
├── api-tls (ingress api)
├── gateway-tls (ingress gateway)
├── mongodb-tls (MongoDB server)
├── elasticsearch-tls (ES server + truststore.jks)
├── api-internal-tls (Jetty API server + keystore.p12)
└── gateway-internal-tls (Vert.x gateway server + keystore.p12)
```
### cert-manager keystores
Three secrets include PKCS12 keystores automatically generated by cert-manager:
- `api-internal-tls` → `keystore.p12` (Jetty server cert) + `truststore.p12` (CA trust for JVM)
- `gateway-internal-tls` → `keystore.p12` (Vert.x server cert) + `truststore.p12` (CA trust for JVM)
- `elasticsearch-tls` → `keystore.p12` + `truststore.p12` (ES server cert — keystore unused by Gravitee)
Each component mounts only its own internal TLS secret at `/run/secrets/tls/`, providing both
the server keystore and the JVM truststore from a single volume.
Keystore password is stored in the `gravitee-jks-password` secret.
### Notable SANs
- `api-internal-tls` includes `api.gravitee.sttlab.pc` — required for nginx → pod TLS verification
- `gateway-internal-tls` includes `gateway.gravitee.sttlab.pc` — same reason
- `elasticsearch-tls` includes `gravitee-es-master.gravitee-apim.svc.cluster.local`
### Browser trust
Export the CA for browser import:
```bash
kubectl get secret gravitee-ca-tls -n gravitee-apim \
-o jsonpath='{.data.ca\.crt}' | base64 -d > gravitee-ca.crt
```
Import `gravitee-ca.crt` into the OS keychain or browser trust store.
---
## Kubernetes secrets
All secrets are created by `secrets-create.sh` before the first deployment.
| Secret | Contents | Consumed by |
|--------|----------|-------------|
| `mongodb-credentials` | root/gravitee MongoDB passwords | Bitnami MongoDB chart |
| `gravitee-mongodb-uri` | `GRAVITEE_MANAGEMENT_MONGODB_URI`, `GRAVITEE_RATELIMIT_MONGODB_URI` | api, gateway (env) |
| `gravitee-jwt` | `GRAVITEE_JWT_SECRET` | api (env) |
| `gravitee-admin` | `admin-password-plain`, `admin-password-bcrypt` | api (env) |
| `gravitee-es-master-credentials` | ES `username`, `password` | api, gateway (env) |
| `gravitee-jks-password` | keystore password | api, gateway (env) + cert-manager |
| `gravitee-ca-trust` | `ca.crt` only — no tls.crt/key | nginx ingress `proxy-ssl-secret` |
> `gravitee-ca-trust` is created by `secrets-create.sh` after the certificates are ready (it reads `ca.crt` from `gravitee-ca-tls`). It must contain **only** `ca.crt` — if `tls.crt`/`tls.key` were present, nginx would present them as a client certificate, triggering unintended mTLS toward backends.
> `gravitee-ca-trust` is a dedicated secret containing only `ca.crt`. The `proxy-ssl-secret` nginx annotation presents `tls.crt`/`tls.key` as a client certificate if they exist, which would unintentionally trigger mTLS toward backends.
### Credential injection
All credentials are injected via `env[].valueFrom.secretKeyRef` (no `envFrom`). The rule is:
- **Non-sensitive config** (endpoints, flags) → `value:` directly in `env`
- **Secrets** (passwords, tokens, URIs with credentials) → `valueFrom.secretKeyRef`
Example from `apim-values.yml`:
```yaml
env:
- name: GRAVITEE_MANAGEMENT_MONGODB_URI # credentials + TLS embedded in URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_MANAGEMENT_MONGODB_URI
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_ENDPOINTS_0 # non-sensitive
value: "https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200"
```
---
## Internal TLS
### MongoDB
TLS enabled via `tls=true` in the connection URI. Server certificate validation is handled by the JVM truststore:
```
JAVA_OPTS: -Djavax.net.ssl.trustStoreType=PKCS12
-Djavax.net.ssl.trustStore=/run/secrets/tls/truststore.p12
-Djavax.net.ssl.trustStorePassword=${JKS_PASSWORD}
```
The `truststore.p12` is sourced from the component's own internal TLS secret (`api-internal-tls` or `gateway-internal-tls`), which cert-manager populates with the issuing CA chain.
MongoDB runs with `--tlsAllowConnectionsWithoutCertificates` (no client mTLS) but requires server TLS (`--tlsMode=requireTLS`).
### Elasticsearch
HTTPS with basic auth. Server certificate signed by the Gravitee CA, validated by the same JVM truststore. Username/password injected via Gravitee property-path env vars:
- `GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Management API
- `GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Gateway
### nginx → backends (proxy TLS)
nginx verifies backend certificates:
```yaml
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name api.gravitee.sttlab.pc;
```
- `proxy-ssl-secret` provides the CA for chain verification
- `proxy_ssl_name` (via snippet) tells nginx which hostname to check against the backend cert SANs
- `proxy_ssl_server_name on` is intentionally **absent**: it modifies the TLS ClientHello in a way that triggers an `illegal parameter` alert in Jetty 12
Snippets require `annotations-risk-level: Critical` in the `ingress-nginx-controller` ConfigMap.
---
## UI ingress (SPA)
The console and portal are Angular SPAs. The Gravitee chart injects `rewrite-target: /$1` by default. Without a capture group in the path, all asset requests are rewritten to `/` (index.html served instead of CSS/JS).
Fix applied:
```yaml
path: /(.*)
pathType: ImplementationSpecific
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
```
---
## Deployment
### Prerequisites
- k3s with nginx ingress and cert-manager installed
- Helm repos added: `bitnami`, `elastic`, `graviteeio`
- `htpasswd` available (for admin BCrypt hash generation)
### First install
```bash
./deploy.sh
```
Or step by step:
```bash
# 1. Create credential secrets
./secrets-create.sh
# 2. Deploy PKI and wait for certificates
kubectl apply -f certificates.yml
kubectl -n gravitee-apim wait --for=condition=Ready certificate --all --timeout=180s
# 3. Create CA trust secret for nginx (requires certificates to be ready first)
kubectl -n gravitee-apim get secret gravitee-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | \
kubectl -n gravitee-apim create secret generic gravitee-ca-trust --from-file=ca.crt=/dev/stdin \
--dry-run=client -o yaml | kubectl apply -f -
# 4. Enable nginx ingress snippets
kubectl patch configmap ingress-nginx-controller -n ingress-nginx \
--type merge -p '{"data":{"allow-snippet-annotations":"true","annotations-risk-level":"Critical"}}'
kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx
# 5. Deploy backends
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
# 6. Deploy Gravitee
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```
### Rebuild workloads only (secrets and certs preserved)
```bash
helm uninstall graviteeio-apim mongodb elasticsearch -n gravitee-apim
kubectl delete pvc -n gravitee-apim --all
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```
> Secrets, certs, PVs and the `gravitee-ca-trust` secret are preserved across workload rebuilds.
### /etc/hosts
```
192.168.1.18 console.gravitee.sttlab.pc portal.gravitee.sttlab.pc api.gravitee.sttlab.pc gateway.gravitee.sttlab.pc
```
---
## Access
| URL | Description |
|-----|-------------|
| https://console.gravitee.sttlab.pc | Management console |
| https://portal.gravitee.sttlab.pc | Developer portal |
| https://api.gravitee.sttlab.pc/management | Management API |
| https://gateway.gravitee.sttlab.pc | Gateway (data plane) |
**Admin credentials:**
```bash
# Username: admin
# Password (plaintext):
kubectl get secret gravitee-admin -n gravitee-apim \
-o jsonpath='{.data.admin-password-plain}' | base64 -d && echo
```
---
## Security notes
- The Gravitee CA is self-signed. Replace with a Vault PKI Issuer in production (see comment in `certificates.yml`).
- `proxy-ssl-verify: "on"` is enabled on all backend ingresses — nginx validates pod certificates.
- Client mTLS is not enabled for MongoDB (complexity vs. benefit for intra-cluster traffic).
- `annotations-risk-level: Critical` on the ingress controller is required for `proxy_ssl_name` snippets — document this in the ops runbook.