infra-gravitee-apim/platform/README.md

# Gravitee APIM — k3s single-node deployment

Production-like deployment of Gravitee APIM OSS on k3s with end-to-end TLS, credentials injected from Kubernetes secrets, and an internal CA managed by cert-manager.

---

## Architecture

```
Internet
  └── nginx ingress (TLS termination + HTTPS re-encryption to backends)
        ├── console.gravitee.sttlab.pc  →  graviteeio-apim-ui (nginx, port 8080)
        ├── portal.gravitee.sttlab.pc   →  graviteeio-apim-portal (nginx, port 8080)
        ├── api.gravitee.sttlab.pc      →  graviteeio-apim-api (Jetty/HTTPS, port 8083)
        └── gateway.gravitee.sttlab.pc  →  graviteeio-apim-gateway (Vert.x/HTTPS, port 8082)

Internal backends:
  graviteeio-apim-api     ──TLS──► mongodb:27017
  graviteeio-apim-api     ──TLS──► gravitee-es-master:9200
  graviteeio-apim-gateway ──TLS──► mongodb:27017
  graviteeio-apim-gateway ──TLS──► gravitee-es-master:9200
```

**Stack:**
- **Gravitee APIM 4.x** — Management API (Jetty), Gateway (Vert.x/Netty), Console UI and Developer Portal (Angular/nginx)
- **Gravitee Kubernetes Operator (GKO)** — GitOps management of APIs via CRDs
- **MongoDB 8.x** (Bitnami) — management and rate limiting persistence
- **Elasticsearch 8.x** (Elastic) — analytics and reporting
- **cert-manager** — internal self-signed PKI with automatic renewal
- **nginx ingress** — external TLS termination + TLS proxy to backends

---

## PKI and certificates

### Trust chain

```
gravitee-selfsigned-bootstrap (Issuer, self-signed)
  └── gravitee-ca (Certificate isCA:true) → secret: gravitee-ca-tls
        └── gravitee-ca-issuer (CA Issuer)
              ├── console-tls          (ingress console)
              ├── portal-tls           (ingress portal)
              ├── api-tls              (ingress api)
              ├── gateway-tls          (ingress gateway)
              ├── mongodb-tls          (MongoDB server)
              ├── elasticsearch-tls    (ES server + truststore.jks)
              ├── api-internal-tls     (Jetty API server + keystore.p12)
              └── gateway-internal-tls (Vert.x gateway server + keystore.p12)
```

### cert-manager keystores

Three secrets include PKCS12 keystores automatically generated by cert-manager:
- `api-internal-tls` → `keystore.p12` (Jetty server cert) + `truststore.p12` (CA trust for JVM)
- `gateway-internal-tls` → `keystore.p12` (Vert.x server cert) + `truststore.p12` (CA trust for JVM)
- `elasticsearch-tls` → `keystore.p12` + `truststore.p12` (ES server cert — keystore unused by Gravitee)

Each component mounts only its own internal TLS secret at `/run/secrets/tls/`, providing both
the server keystore and the JVM truststore from a single volume.

Keystore password is stored in the `gravitee-jks-password` secret.

### Notable SANs

- `api-internal-tls` includes `api.gravitee.sttlab.pc` — required for nginx → pod TLS verification
- `gateway-internal-tls` includes `gateway.gravitee.sttlab.pc` — same reason
- `elasticsearch-tls` includes `gravitee-es-master.gravitee-apim.svc.cluster.local`

### Browser trust

Export the CA for browser import:
```bash
kubectl get secret gravitee-ca-tls -n gravitee-apim \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > gravitee-ca.crt
```
Import `gravitee-ca.crt` into the OS keychain or browser trust store.

---

## Kubernetes secrets

All secrets are created by `secrets-create.sh` before the first deployment.

| Secret | Contents | Consumed by |
|--------|----------|-------------|
| `mongodb-credentials` | root/gravitee MongoDB passwords | Bitnami MongoDB chart |
| `gravitee-mongodb-uri` | `GRAVITEE_MANAGEMENT_MONGODB_URI`, `GRAVITEE_RATELIMIT_MONGODB_URI` | api, gateway (Secret Manager) |
| `gravitee-jwt` | `GRAVITEE_JWT_SECRET` | api (Secret Manager) |
| `gravitee-admin` | `admin-password-plain`, `admin-password-bcrypt` | api (Secret Manager) |
| `gravitee-es-master-credentials` | ES `username`, `password` | api, gateway (Secret Manager) |
| `gravitee-encryption` | `api-properties-encryption-secret` | api, gateway (Secret Manager) |
| `gravitee-jks-password` | keystore password | api, gateway (`JAVA_OPTS` env + Secret Manager for `ssl.keystore.password`) + cert-manager |
| `gravitee-ca-trust` | `ca.crt` only — no tls.crt/key | nginx ingress `proxy-ssl-secret` |

> `gravitee-ca-trust` is created by `secrets-create.sh` after the certificates are ready (it reads `ca.crt` from `gravitee-ca-tls`). It must contain **only** `ca.crt` — if `tls.crt`/`tls.key` were present, nginx would present them as a client certificate, triggering unintended mTLS toward backends.

### Credential injection

Credentials are resolved at runtime by the **Gravitee Secret Manager** (Kubernetes provider, enabled via `secrets.kubernetes`). Helm values reference secrets using the `secret://kubernetes/<secret-name>:<key>` syntax, which is rendered verbatim into `gravitee.yml` and resolved by Gravitee on startup.

```yaml
# In apim-values.yml — resolved by Gravitee Secret Manager at runtime
jwtSecret: "secret://kubernetes/gravitee-jwt:GRAVITEE_JWT_SECRET"
mongo:
  uri: "secret://kubernetes/gravitee-mongodb-uri:GRAVITEE_MANAGEMENT_MONGODB_URI"
es:
  security:
    username: "secret://kubernetes/gravitee-es-master-credentials:username"
    password: "secret://kubernetes/gravitee-es-master-credentials:password"
```

The only secret still injected as a Kubernetes env var is `gravitee-jks-password` (`JKS_PASSWORD`), because it is needed in `JAVA_OPTS` before the JVM starts — before the Gravitee Secret Manager is available.

---

## Internal TLS

### MongoDB

TLS enabled via `tls=true` in the connection URI. Server certificate validation is handled by the JVM truststore:
```
JAVA_OPTS: -Djavax.net.ssl.trustStoreType=PKCS12
           -Djavax.net.ssl.trustStore=/run/secrets/tls/truststore.p12
           -Djavax.net.ssl.trustStorePassword=${JKS_PASSWORD}
```
The `truststore.p12` is sourced from the component's own internal TLS secret (`api-internal-tls` or `gateway-internal-tls`), which cert-manager populates with the issuing CA chain.

MongoDB runs with `--tlsAllowConnectionsWithoutCertificates` (no client mTLS) but requires server TLS (`--tlsMode=requireTLS`).

### Elasticsearch

HTTPS with basic auth. Server certificate signed by the Gravitee CA, validated by the same JVM truststore. The HTTPS endpoint and credentials are configured in `apim-values.yml` via `es.endpoints` / `es.security.*` and resolved at runtime by the Gravitee Secret Manager.

### nginx → backends (proxy TLS)

nginx verifies backend certificates:
```yaml
nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
nginx.ingress.kubernetes.io/proxy-ssl-secret: "gravitee-apim/gravitee-ca-trust"
nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_ssl_name api.gravitee.sttlab.pc;
```

- `proxy-ssl-secret` provides the CA for chain verification
- `proxy_ssl_name` (via snippet) tells nginx which hostname to check against the backend cert SANs
- `proxy_ssl_server_name on` is intentionally **absent**: it modifies the TLS ClientHello in a way that triggers an `illegal parameter` alert in Jetty 12

Snippets require `annotations-risk-level: Critical` in the `ingress-nginx-controller` ConfigMap.

---

## UI ingress (SPA)

The console and portal are Angular SPAs. The Gravitee chart injects `rewrite-target: /$1` by default. Without a capture group in the path, all asset requests are rewritten to `/` (index.html served instead of CSS/JS).

Fix applied:
```yaml
path: /(.*)
pathType: ImplementationSpecific
annotations:
  nginx.ingress.kubernetes.io/use-regex: "true"
  nginx.ingress.kubernetes.io/rewrite-target: /$1
```

---

## Deployment

### Prerequisites

- k3s with nginx ingress and cert-manager installed
- Helm repos added: `bitnami`, `elastic`, `graviteeio`
- `htpasswd` available (for admin BCrypt hash generation)

### First install

```bash
./deploy.sh
```

Or step by step:

```bash
# 1. Create credential secrets
./secrets-create.sh

# 2. Deploy PKI and wait for certificates
kubectl apply -f certificates.yml
kubectl -n gravitee-apim wait --for=condition=Ready certificate --all --timeout=180s

# 3. Create CA trust secret for nginx (requires certificates to be ready first)
kubectl -n gravitee-apim get secret gravitee-ca-tls -o jsonpath='{.data.ca\.crt}' | base64 -d | \
  kubectl -n gravitee-apim create secret generic gravitee-ca-trust --from-file=ca.crt=/dev/stdin \
  --dry-run=client -o yaml | kubectl apply -f -

# 4. Enable nginx ingress snippets
kubectl patch configmap ingress-nginx-controller -n ingress-nginx \
  --type merge -p '{"data":{"allow-snippet-annotations":"true","annotations-risk-level":"Critical"}}'
kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx

# 5. Deploy backends
helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml

# 6. Deploy Gravitee
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml

# 7. Install GKO (after APIM is ready)
# Create GKO credentials secret first (see GKO section below)
helm upgrade --install gko graviteeio/gko \
  --namespace gravitee-apim \
  --set manager.httpClient.trustStore.path=/etc/ssl/certs/gravitee-ca.crt \
  --set-json 'manager.volumes=[{"name":"gravitee-ca","secret":{"secretName":"gravitee-ca-tls","items":[{"key":"ca.crt","path":"gravitee-ca.crt"}]}}]' \
  --set-json 'manager.volumeMounts=[{"name":"gravitee-ca","mountPath":"/etc/ssl/certs/gravitee-ca.crt","subPath":"gravitee-ca.crt","readOnly":true}]'

# 8. Apply GKO resources
kubectl apply -f gko-management-context.yaml
kubectl apply -f gko-api.yaml
```

### Rebuild workloads only (secrets and certs preserved)

```bash
helm uninstall graviteeio-apim mongodb elasticsearch -n gravitee-apim
kubectl delete pvc -n gravitee-apim --all

helm upgrade --install mongodb bitnami/mongodb -n gravitee-apim -f mongo-values.yml
helm upgrade --install elasticsearch elastic/elasticsearch -n gravitee-apim -f elastic-values.yml
helm upgrade --install graviteeio-apim graviteeio/apim -n gravitee-apim -f apim-values.yml
```

> Secrets, certs, PVs and the `gravitee-ca-trust` secret are preserved across workload rebuilds.

### /etc/hosts

```
192.168.1.18  console.gravitee.sttlab.pc portal.gravitee.sttlab.pc api.gravitee.sttlab.pc gateway.gravitee.sttlab.pc
```

---

## Access

| URL | Description |
|-----|-------------|
| https://console.gravitee.sttlab.pc | Management console |
| https://portal.gravitee.sttlab.pc | Developer portal |
| https://api.gravitee.sttlab.pc/management | Management API |
| https://gateway.gravitee.sttlab.pc | Gateway (data plane) |

**Admin credentials:**
```bash
# Username: admin
# Password (plaintext):
kubectl get secret gravitee-admin -n gravitee-apim \
  -o jsonpath='{.data.admin-password-plain}' | base64 -d && echo
```

---

## Gravitee Kubernetes Operator (GKO)

GKO enables GitOps-driven API management: APIs, plans and subscriptions are declared as Kubernetes CRDs and automatically synced to the Management API.

### Installation

```bash
helm upgrade --install gko graviteeio/gko \
  --namespace gravitee-apim \
  --set manager.httpClient.trustStore.path=/etc/ssl/certs/gravitee-ca.crt \
  --set-json 'manager.volumes=[{"name":"gravitee-ca","secret":{"secretName":"gravitee-ca-tls","items":[{"key":"ca.crt","path":"gravitee-ca.crt"}]}}]' \
  --set-json 'manager.volumeMounts=[{"name":"gravitee-ca","mountPath":"/etc/ssl/certs/gravitee-ca.crt","subPath":"gravitee-ca.crt","readOnly":true}]'
```

The Gravitee CA is mounted into the GKO pod so its HTTP client trusts the Management API's certificate (port 83, HTTPS). Without this, GKO would fail to connect when validating or reconciling CRDs.

### GKO secrets

| Secret | Contents | Purpose |
|--------|----------|---------|
| `gko-management-credentials` | `username`, `password` | GKO → Management API auth via `ManagementContext.spec.auth.secretRef` |

Create after deploying APIM:
```bash
ADMIN_PLAIN=$(kubectl get secret gravitee-admin -n gravitee-apim \
  -o jsonpath='{.data.admin-password-plain}' | base64 -d)
kubectl create secret generic gko-management-credentials -n gravitee-apim \
  --from-literal=username=admin \
  --from-literal=password="${ADMIN_PLAIN}"
```

### CRDs

| CRD | Description |
|-----|-------------|
| `ManagementContext` | Connection to a Gravitee Management API instance |
| `ApiV4Definition` | Gravitee v4 API (proxy, message, native) |
| `ApiDefinition` | Gravitee v2 API (legacy) |
| `Application` | Consumer application |
| `ApiResource` | Shared resource (cache, OAuth2, etc.) |
| `SharedPolicyGroup` | Reusable policy group |
| `Subscription` | API subscription |

### ManagementContext

`gko-management-context.yaml` — connects GKO to the local Management API:
```yaml
spec:
  baseUrl: https://graviteeio-apim-api.gravitee-apim.svc.cluster.local:83
  environmentId: DEFAULT
  organizationId: DEFAULT
  auth:
    secretRef:
      name: gko-management-credentials
      namespace: gravitee-apim
```

> Port 83 is the Kubernetes service port (maps to container port 8083).

### Deploying an API

```bash
kubectl apply -f gko-management-context.yaml
kubectl apply -f gko-api.yaml
kubectl get apiv4definition -n gravitee-apim
```

The `processingStatus: Completed` and `state: STARTED` in the status confirm the API is live on the gateway.

---

## Security notes

- The Gravitee CA is self-signed. Replace with a Vault PKI Issuer in production (see comment in `certificates.yml`).
- `proxy-ssl-verify: "on"` is enabled on all backend ingresses — nginx validates pod certificates.
- Client mTLS is not enabled for MongoDB (complexity vs. benefit for intra-cluster traffic).
- `annotations-risk-level: Critical` on the ingress controller is required for `proxy_ssl_name` snippets — document this in the ops runbook.