stt/infra-gravitee-apim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Simplification of Mongo and ES config

Browse Source
This commit is contained in:
sttlab
2026-05-03 12:38:46 +00:00
parent 0d1e5dd9b8
commit 5a48150820
2 changed files with 288 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+233
View File
@@ -0,0 +1,233 @@
# Gravitee APIM — k3s single-node deployment
Production-like deployment of Gravitee APIM OSS on k3s with end-to-end TLS, credentials injected from Kubernetes secrets, and an internal CA managed by cert-manager.
---
## Architecture
```
Internet
└── nginx ingress (TLS termination + HTTPS re-encryption to backends)
├── console.gravitee.sttlab.pc → graviteeio-apim-ui (nginx, port 8080)
├── portal.gravitee.sttlab.pc → graviteeio-apim-portal (nginx, port 8080)
├── api.gravitee.sttlab.pc → graviteeio-apim-api (Jetty/HTTPS, port 8083)
└── gateway.gravitee.sttlab.pc → graviteeio-apim-gateway (Vert.x/HTTPS, port 8082)
Internal backends:
graviteeio-apim-api ──TLS──► mongodb:27017
graviteeio-apim-api ──TLS──► gravitee-es-master:9200
graviteeio-apim-gateway ──TLS──► mongodb:27017
graviteeio-apim-gateway ──TLS──► gravitee-es-master:9200
```
**Stack:**
- **Gravitee APIM 4.x** — Management API (Jetty), Gateway (Vert.x/Netty), Console UI and Developer Portal (Angular/nginx)
- **MongoDB 8.x** (Bitnami) — management and rate limiting persistence
- **Elasticsearch 8.x** (Elastic) — analytics and reporting
- **cert-manager** — internal self-signed PKI with automatic renewal
- **nginx ingress** — external TLS termination + TLS proxy to backends
---
## PKI and certificates
### Trust chain
```
gravitee-selfsigned-bootstrap (Issuer, self-signed)
└── gravitee-ca (Certificate isCA:true) → secret: gravitee-ca-tls
└── gravitee-ca-issuer (CA Issuer)
├── console-tls (ingress console)
├── portal-tls (ingress portal)
├── api-tls (ingress api)
├── gateway-tls (ingress gateway)
├── mongodb-tls (MongoDB server)
├── elasticsearch-tls (ES server + truststore.jks)
├── api-internal-tls (Jetty API server + keystore.p12)
└── gateway-internal-tls (Vert.x gateway server + keystore.p12)
```
### cert-manager keystores
Two secrets include PKCS12 keystores automatically generated by cert-manager:
- `api-internal-tls` → `keystore.p12` (used by Jetty)
- `gateway-internal-tls` → `keystore.p12` (used by Vert.x)
- `elasticsearch-tls` → `truststore.jks` (used by the JVM to trust the ES certificate)
Keystore password is stored in the `gravitee-jks-password` secret.
### Notable SANs
- `api-internal-tls` includes `api.gravitee.sttlab.pc` — required for nginx → pod TLS verification
- `gateway-internal-tls` includes `gateway.gravitee.sttlab.pc` — same reason
- `elasticsearch-tls` includes `gravitee-es-master.gravitee-apim.svc.cluster.local`
### Browser trust
Export the CA for browser import:
```bash
kubectl get secret gravitee-ca-tls -n gravitee-apim \
-o jsonpath='{.data.ca\.crt}' | base64 -d > gravitee-ca.crt
```
Import `gravitee-ca.crt` into the OS keychain or browser trust store.
---
## Kubernetes secrets
All secrets are created by `secrets-create.sh` before the first deployment.
| Secret | Contents | Consumed by |
|--------|----------|-------------|
| `mongodb-credentials` | root/gravitee MongoDB passwords | Bitnami MongoDB chart |
| `gravitee-mongodb-uri` | `GRAVITEE_MANAGEMENT_MONGODB_URI`, `GRAVITEE_RATELIMIT_MONGODB_URI` | api, gateway (env) |
| `gravitee-jwt` | `GRAVITEE_JWT_SECRET` | api (env) |
| `gravitee-admin` | `admin-password-plain`, `admin-password-bcrypt` | api (env) |
| `gravitee-es-master-credentials` | ES `username`, `password` | api, gateway (env) |
| `gravitee-jks-password` | keystore password | api, gateway (env) + cert-manager |
| `gravitee-ca-trust` | `ca.crt` only — no tls.crt/key | nginx ingress proxy-ssl-secret |
> `gravitee-ca-trust` is a dedicated secret containing only `ca.crt`. The `proxy-ssl-secret` nginx annotation presents `tls.crt`/`tls.key` as a client certificate if they exist, which would unintentionally trigger mTLS toward backends.
### Credential injection
All credentials are injected via `env[].valueFrom.secretKeyRef` (no `envFrom`). The rule is:
- **Non-sensitive config** (endpoints, flags) → `value:` directly in `env`
- **Secrets** (passwords, tokens, URIs with credentials) → `valueFrom.secretKeyRef`
Example from `apim-values.yml`:
```yaml
env:
- name: GRAVITEE_MANAGEMENT_MONGODB_URI # credentials + TLS embedded in URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_MANAGEMENT_MONGODB_URI
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_ENDPOINTS_0 # non-sensitive
value: "https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200"
```
---
## Internal TLS
### MongoDB
TLS enabled via `tls=true` in the connection URI. Server certificate validation is handled by the JVM truststore:
```
JAVA_OPTS: -Djavax.net.ssl.trustStore=/run/secrets/truststore/truststore.jks
-Djavax.net.ssl.trustStorePassword=${JKS_PASSWORD}
```
The `truststore.jks` is sourced from the `elasticsearch-tls` secret (contains the Gravitee CA).
MongoDB runs with `--tlsAllowConnectionsWithoutCertificates` (no client mTLS) but requires server TLS (`--tlsMode=requireTLS`).
### Elasticsearch
HTTPS with basic auth. Server certificate signed by the Gravitee CA, validated by the same JVM truststore. Username/password injected via Gravitee property-path env vars:
- `GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Management API
- `GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Gateway
### nginx → backends (proxy TLS)
nginx verifies backend certificates:
```yaml
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name api.gravitee.sttlab.pc;
```
- `proxy-ssl-secret` provides the CA for chain verification
- `proxy_ssl_name` (via snippet) tells nginx which hostname to check against the backend cert SANs
- `proxy_ssl_server_name on` is intentionally **absent**: it modifies the TLS ClientHello in a way that triggers an `illegal parameter` alert in Jetty 12
Snippets require `annotations-risk-level: Critical` in the `ingress-nginx-controller` ConfigMap.
---
## UI ingress (SPA)
The console and portal are Angular SPAs. The Gravitee chart injects `rewrite-target: /$1` by default. Without a capture group in the path, all asset requests are rewritten to `/` (index.html served instead of CSS/JS).
Fix applied:
```yaml
path: /(.*)
pathType: ImplementationSpecific
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
```
---
## Deployment
### Prerequisites
- k3s with nginx ingress and cert-manager installed
- Helm repos added: `bitnami`, `elastic`, `graviteeio`
- `htpasswd` available (for admin BCrypt hash generation)
### First install
```bash
# 1. Create secrets
./secrets-create.sh
# 2. Deploy PKI
kubectl apply -f certificates.yml
# 3. Deploy backends
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
# 4. Deploy Gravitee
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```
### Rebuild workloads only (secrets and certs preserved)
```bash
helm uninstall graviteeio-apim mongodb elasticsearch -n gravitee-apim
kubectl delete pvc -n gravitee-apim --all
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```
### /etc/hosts
```
192.168.1.18 console.gravitee.sttlab.pc portal.gravitee.sttlab.pc api.gravitee.sttlab.pc gateway.gravitee.sttlab.pc
```
---
## Access
| URL | Description |
|-----|-------------|
| https://console.gravitee.sttlab.pc | Management console |
| https://portal.gravitee.sttlab.pc | Developer portal |
| https://api.gravitee.sttlab.pc/management | Management API |
| https://gateway.gravitee.sttlab.pc | Gateway (data plane) |
**Admin credentials:**
```bash
# Username: admin
# Password (plaintext):
kubectl get secret gravitee-admin -n gravitee-apim \
-o jsonpath='{.data.admin-password-plain}' | base64 -d && echo
```
---
## Security notes
- The Gravitee CA is self-signed. Replace with a Vault PKI Issuer in production (see comment in `certificates.yml`).
- `proxy-ssl-verify: "on"` is enabled on all backend ingresses — nginx validates pod certificates.
- Client mTLS is not enabled for MongoDB (complexity vs. benefit for intra-cluster traffic).
- `annotations-risk-level: Critical` on the ingress controller is required for `proxy_ssl_name` snippets — document this in the ops runbook.
apim-values.yml
+55 -34
View File
@@ -7,24 +7,6 @@
adminAccountEnable: true
adminPasswordBcrypt: "${GRAVITEE_ADMIN_PASSWORD_BCRYPT}"
# External MongoDB — URI injected at runtime via GRAVITEE_MANAGEMENT/RATELIMIT_MONGODB_URI
# from the gravitee-mongodb-uri secret (see deployment.envFrom below)
mongo:
dbhost: mongodb.gravitee-apim.svc.cluster.local
dbname: gravitee
dbport: 27017
rsEnabled: false
# External Elasticsearch (HTTPS + basic auth)
# Password injected at runtime via env var from gravitee-es-master-credentials secret
es:
endpoints:
- https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200
security:
enabled: true
username: elastic
password: ""
# ============================================================
# API Gateway (data plane) - 2 replicas
# ============================================================
@@ -60,12 +42,26 @@ gateway:
mountPath: /run/secrets/truststore
readOnly: true
deployment:
envFrom:
- secretRef:
name: gravitee-mongodb-uri
env:
- name: GRAVITEE_MANAGEMENT_MONGODB_URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_MANAGEMENT_MONGODB_URI
- name: GRAVITEE_RATELIMIT_MONGODB_URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_RATELIMIT_MONGODB_URI
- name: GRAVITEE_REPORTERS_ELASTICSEARCH_ENDPOINTS_0
value: "https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200"
- name: GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_ENABLED
value: "true"
- name: GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_USERNAME
valueFrom:
secretKeyRef:
name: gravitee-es-master-credentials
key: username
- name: GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_PASSWORD
valueFrom:
secretKeyRef:
@@ -104,9 +100,11 @@ gateway:
enabled: true
ingressClassName: nginx
annotations:
# Gateway already terminates TLS internally; nginx forwards as HTTPS
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name gateway.gravitee.sttlab.pc;
hosts:
- gateway.gravitee.sttlab.pc
path: /
@@ -153,19 +151,36 @@ api:
mountPath: /run/secrets/truststore
readOnly: true
deployment:
envFrom:
- secretRef:
name: gravitee-mongodb-uri
- secretRef:
name: gravitee-jwt
env:
- name: GRAVITEE_MANAGEMENT_MONGODB_URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_MANAGEMENT_MONGODB_URI
- name: GRAVITEE_RATELIMIT_MONGODB_URI
valueFrom:
secretKeyRef:
name: gravitee-mongodb-uri
key: GRAVITEE_RATELIMIT_MONGODB_URI
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_ENDPOINTS_0
value: "https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200"
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_ENABLED
value: "true"
- name: GRAVITEE_JWT_SECRET
valueFrom:
secretKeyRef:
name: gravitee-jwt
key: GRAVITEE_JWT_SECRET
- name: GRAVITEE_ADMIN_PASSWORD_BCRYPT
valueFrom:
secretKeyRef:
name: gravitee-admin
key: admin-password-bcrypt
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_USERNAME
valueFrom:
secretKeyRef:
name: gravitee-es-master-credentials
key: username
- name: GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_PASSWORD
valueFrom:
secretKeyRef:
@@ -209,7 +224,10 @@ api:
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name api.gravitee.sttlab.pc;
path: /management
pathType: Prefix
hosts:
@@ -223,7 +241,10 @@ api:
ingressClassName: nginx
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_ssl_name api.gravitee.sttlab.pc;
path: /portal
pathType: Prefix
hosts: