# Gravitee APIM — k3s single-node deployment

Production-like deployment of Gravitee APIM OSS on k3s with end-to-end TLS, credentials injected from Kubernetes secrets, and an internal CA managed by cert-manager.

---

## Architecture

```
Internet
  └── nginx ingress (TLS termination + HTTPS re-encryption to backends)
        ├── console.gravitee.sttlab.pc  →  graviteeio-apim-ui (nginx, port 8080)
        ├── portal.gravitee.sttlab.pc   →  graviteeio-apim-portal (nginx, port 8080)
        ├── api.gravitee.sttlab.pc      →  graviteeio-apim-api (Jetty/HTTPS, port 8083)
        └── gateway.gravitee.sttlab.pc  →  graviteeio-apim-gateway (Vert.x/HTTPS, port 8082)

Internal backends:
  graviteeio-apim-api     ──TLS──► mongodb:27017
  graviteeio-apim-api     ──TLS──► gravitee-es-master:9200
  graviteeio-apim-gateway ──TLS──► mongodb:27017
  graviteeio-apim-gateway ──TLS──► gravitee-es-master:9200
```

**Stack:**
- **Gravitee APIM 4.x** — Management API (Jetty), Gateway (Vert.x/Netty), Console UI and Developer Portal (Angular/nginx)
- **MongoDB 8.x** (Bitnami) — management and rate limiting persistence
- **Elasticsearch 8.x** (Elastic) — analytics and reporting
- **cert-manager** — internal self-signed PKI with automatic renewal
- **nginx ingress** — external TLS termination + TLS proxy to backends

---

## PKI and certificates

### Trust chain

```
gravitee-selfsigned-bootstrap (Issuer, self-signed)
  └── gravitee-ca (Certificate isCA:true) → secret: gravitee-ca-tls
        └── gravitee-ca-issuer (CA Issuer)
              ├── console-tls          (ingress console)
              ├── portal-tls           (ingress portal)
              ├── api-tls              (ingress api)
              ├── gateway-tls          (ingress gateway)
              ├── mongodb-tls          (MongoDB server)
              ├── elasticsearch-tls    (ES server + truststore.jks)
              ├── api-internal-tls     (Jetty API server + keystore.p12)
              └── gateway-internal-tls (Vert.x gateway server + keystore.p12)
```

### cert-manager keystores

Three secrets include PKCS12 keystores automatically generated by cert-manager:
- `api-internal-tls` → `keystore.p12` (Jetty server cert) + `truststore.p12` (CA trust for JVM)
- `gateway-internal-tls` → `keystore.p12` (Vert.x server cert) + `truststore.p12` (CA trust for JVM)
- `elasticsearch-tls` → `keystore.p12` + `truststore.p12` (ES server cert — keystore unused by Gravitee)

Each component mounts only its own internal TLS secret at `/run/secrets/tls/`, providing both
the server keystore and the JVM truststore from a single volume.

Keystore password is stored in the `gravitee-jks-password` secret.

### Notable SANs

- `api-internal-tls` includes `api.gravitee.sttlab.pc` — required for nginx → pod TLS verification
- `gateway-internal-tls` includes `gateway.gravitee.sttlab.pc` — same reason
- `elasticsearch-tls` includes `gravitee-es-master.gravitee-apim.svc.cluster.local`

### Browser trust

Export the CA for browser import:
```bash
kubectl get secret gravitee-ca-tls -n gravitee-apim \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > gravitee-ca.crt
```
Import `gravitee-ca.crt` into the OS keychain or browser trust store.

---

## Kubernetes secrets

All secrets are created by `secrets-create.sh` before the first deployment.

| Secret | Contents | Consumed by |
|--------|----------|-------------|
| `mongodb-credentials` | root/gravitee MongoDB passwords | Bitnami MongoDB chart |
| `gravitee-mongodb-uri` | `GRAVITEE_MANAGEMENT_MONGODB_URI`, `GRAVITEE_RATELIMIT_MONGODB_URI` | api, gateway (env) |
| `gravitee-jwt` | `GRAVITEE_JWT_SECRET` | api (env) |
| `gravitee-admin` | `admin-password-plain`, `admin-password-bcrypt` | api (env) |
| `gravitee-es-master-credentials` | ES `username`, `password` | api, gateway (env) |
| `gravitee-jks-password` | keystore password | api, gateway (env) + cert-manager |
| `gravitee-ca-trust` | `ca.crt` only — no tls.crt/key | nginx ingress `proxy-ssl-secret` |

> `gravitee-ca-trust` is created by `secrets-create.sh` after the certificates are ready (it reads `ca.crt` from `gravitee-ca-tls`). It must contain **only** `ca.crt` — if `tls.crt`/`tls.key` were present, nginx would present them as a client certificate, triggering unintended mTLS toward backends.

> `gravitee-ca-trust` is a dedicated secret containing only `ca.crt`. The `proxy-ssl-secret` nginx annotation presents `tls.crt`/`tls.key` as a client certificate if they exist, which would unintentionally trigger mTLS toward backends.

### Credential injection

All credentials are injected via `env[].valueFrom.secretKeyRef` (no `envFrom`). The rule is:
- **Non-sensitive config** (endpoints, flags) → `value:` directly in `env`
- **Secrets** (passwords, tokens, URIs with credentials) → `valueFrom.secretKeyRef`

Example from `apim-values.yml`:
```yaml
env:
  - name: GRAVITEE_MANAGEMENT_MONGODB_URI       # credentials + TLS embedded in URI
    valueFrom:
      secretKeyRef:
        name: gravitee-mongodb-uri
        key: GRAVITEE_MANAGEMENT_MONGODB_URI
  - name: GRAVITEE_ANALYTICS_ELASTICSEARCH_ENDPOINTS_0   # non-sensitive
    value: "https://gravitee-es-master.gravitee-apim.svc.cluster.local:9200"
```

---

## Internal TLS

### MongoDB

TLS enabled via `tls=true` in the connection URI. Server certificate validation is handled by the JVM truststore:
```
JAVA_OPTS: -Djavax.net.ssl.trustStoreType=PKCS12
           -Djavax.net.ssl.trustStore=/run/secrets/tls/truststore.p12
           -Djavax.net.ssl.trustStorePassword=${JKS_PASSWORD}
```
The `truststore.p12` is sourced from the component's own internal TLS secret (`api-internal-tls` or `gateway-internal-tls`), which cert-manager populates with the issuing CA chain.

MongoDB runs with `--tlsAllowConnectionsWithoutCertificates` (no client mTLS) but requires server TLS (`--tlsMode=requireTLS`).

### Elasticsearch

HTTPS with basic auth. Server certificate signed by the Gravitee CA, validated by the same JVM truststore. Username/password injected via Gravitee property-path env vars:
- `GRAVITEE_ANALYTICS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Management API
- `GRAVITEE_REPORTERS_ELASTICSEARCH_SECURITY_USERNAME/PASSWORD` → Gateway

### nginx → backends (proxy TLS)

nginx verifies backend certificates:
```yaml
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_name api.gravitee.sttlab.pc;
```

- `proxy-ssl-secret` provides the CA for chain verification
- `proxy_ssl_name` (via snippet) tells nginx which hostname to check against the backend cert SANs
- `proxy_ssl_server_name on` is intentionally **absent**: it modifies the TLS ClientHello in a way that triggers an `illegal parameter` alert in Jetty 12

Snippets require `annotations-risk-level: Critical` in the `ingress-nginx-controller` ConfigMap.

---

## UI ingress (SPA)

The console and portal are Angular SPAs. The Gravitee chart injects `rewrite-target: /$1` by default. Without a capture group in the path, all asset requests are rewritten to `/` (index.html served instead of CSS/JS).

Fix applied:
```yaml
path: /(.*)
pathType: ImplementationSpecific
annotations:
  nginx.ingress.kubernetes.io/use-regex: "true"
  nginx.ingress.kubernetes.io/rewrite-target: /$1
```

---

## Deployment

### Prerequisites

- k3s with nginx ingress and cert-manager installed
- Helm repos added: `bitnami`, `elastic`, `graviteeio`
- `htpasswd` available (for admin BCrypt hash generation)

### First install

```bash
./deploy.sh
```

Or step by step:

```bash
# 1. Create credential secrets
./secrets-create.sh

# 2. Deploy PKI and wait for certificates
kubectl apply -f certificates.yml
kubectl -n gravitee-apim wait --for=condition=Ready certificate --all --timeout=180s

# 3. Create CA trust secret for nginx (requires certificates to be ready first)
kubectl -n gravitee-apim get secret gravitee-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | \
  kubectl -n gravitee-apim create secret generic gravitee-ca-trust --from-file=ca.crt=/dev/stdin \
  --dry-run=client -o yaml | kubectl apply -f -

# 4. Enable nginx ingress snippets
kubectl patch configmap ingress-nginx-controller -n ingress-nginx \
  --type merge -p '{"data":{"allow-snippet-annotations":"true","annotations-risk-level":"Critical"}}'
kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx

# 5. Deploy backends
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml

# 6. Deploy Gravitee
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```

### Rebuild workloads only (secrets and certs preserved)

```bash
helm uninstall graviteeio-apim mongodb elasticsearch -n gravitee-apim
kubectl delete pvc -n gravitee-apim --all

helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```

> Secrets, certs, PVs and the `gravitee-ca-trust` secret are preserved across workload rebuilds.

### /etc/hosts

```
192.168.1.18  console.gravitee.sttlab.pc portal.gravitee.sttlab.pc api.gravitee.sttlab.pc gateway.gravitee.sttlab.pc
```

---

## Access

| URL | Description |
|-----|-------------|
| https://console.gravitee.sttlab.pc | Management console |
| https://portal.gravitee.sttlab.pc | Developer portal |
| https://api.gravitee.sttlab.pc/management | Management API |
| https://gateway.gravitee.sttlab.pc | Gateway (data plane) |

**Admin credentials:**
```bash
# Username: admin
# Password (plaintext):
kubectl get secret gravitee-admin -n gravitee-apim \
  -o jsonpath='{.data.admin-password-plain}' | base64 -d && echo
```

---

## Security notes

- The Gravitee CA is self-signed. Replace with a Vault PKI Issuer in production (see comment in `certificates.yml`).
- `proxy-ssl-verify: "on"` is enabled on all backend ingresses — nginx validates pod certificates.
- Client mTLS is not enabled for MongoDB (complexity vs. benefit for intra-cluster traffic).
- `annotations-risk-level: Critical` on the ingress controller is required for `proxy_ssl_name` snippets — document this in the ops runbook.