Stéphane Tailland
979c9da6c0
The README described the cluster as it was before events existed: no declared topics, no consumer, and no way for another namespace to reach a password Strimzi keeps here. The events themselves are specified by their producer, not here — this repo owns the cluster, not the contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
332 lines
10 KiB
Markdown
332 lines
10 KiB
Markdown
# Kafka on Kubernetes (Strimzi)
|
||
|
||
Single-broker Kafka cluster for dev/test, deployed via the [Strimzi](https://strimzi.io/) operator on a single-node k3s cluster.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
kafka namespace
|
||
├── strimzi-cluster-operator Strimzi operator — reconciles Kafka/KafkaUser/KafkaTopic CRs
|
||
├── kafka-dual-role-0 Kafka 4.3.1 broker (KRaft, controller + broker in one pod)
|
||
└── kafka-entity-operator Topic Operator + User Operator (manages topics and users)
|
||
```
|
||
|
||
### Key design decisions
|
||
|
||
| Choice | Rationale |
|
||
|---|---|
|
||
| **KRaft mode** | No ZooKeeper dependency — simpler, fewer pods |
|
||
| **Single node pool** `dual-role` | One pod acts as both controller and broker, suitable for dev/test |
|
||
| **SASL_SSL + SCRAM-SHA-512** | Authenticated and encrypted connections; TLS managed by Strimzi CA |
|
||
| **Simple authorization** | ACLs enforced per user; `kafka-admin` is declared super user |
|
||
| **`local-path` storage (10 Gi)** | Default k3s storage class; PVC deleted on cluster teardown |
|
||
|
||
### Listeners
|
||
|
||
| Name | Port | Protocol | Auth |
|
||
|---|---|---|---|
|
||
| `tls` | 9093 | SASL_SSL | SCRAM-SHA-512 |
|
||
|
||
Bootstrap address (cluster-internal):
|
||
```
|
||
kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
|
||
```
|
||
|
||
### Users
|
||
|
||
| KafkaUser | Secret | Role |
|
||
|---|---|---|
|
||
| `kafka-admin` | `kafka-admin` | Super user — unrestricted access |
|
||
| `kafka-client` | `kafka-client` | Application user — Read/Write all topics and groups |
|
||
| `classic-models-api` | `classic-models-api` | Producer for `classic-models.*` — **Write and Describe only**, no Read |
|
||
| `classic-models-agent-approvisionnement` | same name | Consumer of `classic-models.catalog` and group prefix `approvisionnement-` — **Read and Describe only**, no Write |
|
||
|
||
Each application account is scoped to what it actually does. The producer
|
||
publishes domain events and consumes nothing, so it is write-only: one that
|
||
could also read every topic exfiltrates every other domain's events if it is
|
||
ever compromised. The consumer is the mirror — it cannot see sales or payments,
|
||
and it cannot write, so a compromised workflow can raise no event that another
|
||
agent would then act on.
|
||
|
||
Duplicate one of these per application. `kafka-client` is the wide-open
|
||
template, useful for a smoke test and for nothing that runs unattended.
|
||
|
||
Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:
|
||
- `password` — the SCRAM password
|
||
- `sasl.jaas.config` — the ready-to-use JAAS config string
|
||
|
||
### Topics
|
||
|
||
Declared in `kafka-topics.yaml`, never auto-created — an application account has
|
||
no `Create` ACL, so a producer would simply be denied, and an auto-created topic
|
||
takes broker defaults nobody chose.
|
||
|
||
| Topic | Partitions | Replicas | Retention |
|
||
|---|---|---|---|
|
||
| `classic-models.catalog` | 3 | 1 | 7 days |
|
||
| `classic-models.sales` | 3 | 1 | 7 days |
|
||
| `classic-models.payments` | 3 | 1 | 7 days |
|
||
|
||
Partition count is the ceiling on consumer parallelism and **cannot be lowered
|
||
later**, so it is set for the traffic expected rather than the traffic today.
|
||
|
||
The events these carry are specified by their producer, in
|
||
`classic-models-api/docs/DOMAIN_EVENTS.md`. This repo owns the cluster, not the
|
||
contract.
|
||
|
||
### Getting credentials into another namespace
|
||
|
||
A Secret cannot be mounted across namespaces, and Strimzi writes a KafkaUser's
|
||
password into **this** one. `credentials-access.yaml` therefore declares a
|
||
`ClusterSecretStore` named `kafka-credentials` (External Secrets, `kubernetes`
|
||
provider) backed by a ServiceAccount that can read Secrets here and nowhere
|
||
else. An application namespace then pulls its own password and the cluster CA
|
||
with an `ExternalSecret`:
|
||
|
||
```yaml
|
||
secretStoreRef:
|
||
kind: ClusterSecretStore
|
||
name: kafka-credentials
|
||
data:
|
||
- secretKey: password
|
||
remoteRef: {key: <kafkauser-name>, property: password}
|
||
- secretKey: ca.crt
|
||
remoteRef: {key: kafka-cluster-ca-cert, property: ca.crt}
|
||
```
|
||
|
||
The password stays generated and rotatable by Strimzi; the copy follows. The
|
||
alternative — copying it by hand — drifts the first time the user is
|
||
regenerated.
|
||
|
||
**The trade, stated:** any namespace allowed to create an `ExternalSecret`
|
||
against this store can pull **any** Kafka account's credentials. Acceptable with
|
||
one operator; on a shared cluster, narrow the Role with `resourceNames` and run
|
||
one store per application.
|
||
|
||
### TLS
|
||
|
||
Strimzi generates and manages its own internal CA. The cluster CA certificate is stored in:
|
||
```
|
||
secret/kafka-cluster-ca-cert (key: ca.crt)
|
||
```
|
||
|
||
This certificate must be trusted by any Kafka client connecting over TLS.
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- `kubectl` configured against the target cluster
|
||
- `helm` v3
|
||
- `make`
|
||
- `cert-manager` is **not** required — Strimzi manages its own CA
|
||
|
||
---
|
||
|
||
## Makefile
|
||
|
||
All operations are available via `make`. Run `make help` for the full reference.
|
||
|
||
```
|
||
Cluster lifecycle:
|
||
deploy Install Strimzi operator + Kafka cluster + users
|
||
smoke-test Run produce/consume end-to-end test
|
||
upgrade Upgrade Kafka (KAFKA_VER=) or Strimzi operator
|
||
teardown Delete the Kafka cluster and namespace
|
||
|
||
Topics:
|
||
topic-list List all topics
|
||
topic-create Create TOPIC (idempotent)
|
||
topic-delete Delete TOPIC
|
||
|
||
Messages:
|
||
produce Produce MESSAGE into TOPIC
|
||
consume Consume MAX messages from TOPIC (from latest)
|
||
list-messages Consume MAX messages from TOPIC (from beginning)
|
||
|
||
Consumer groups:
|
||
group-list List all consumer groups
|
||
group-offsets Show offsets and lag for GROUP
|
||
```
|
||
|
||
Key variables (all overridable on the command line):
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `TOPIC` | `smoke-test` | Topic name |
|
||
| `MESSAGE` | `hello` | Message to produce |
|
||
| `GROUP` | `my-group` | Consumer group |
|
||
| `MAX` | `10` | Max messages to consume |
|
||
| `PARTITIONS` | `1` | Partitions for topic-create |
|
||
| `REPLICAS` | `1` | Replication factor for topic-create |
|
||
| `KAFKA_VER` | `4.1.0` | Kafka version for upgrade |
|
||
| `METADATA_VER` | `4.1-IV0` | KRaft metadata version for upgrade |
|
||
|
||
---
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
make deploy
|
||
```
|
||
|
||
The target performs the following steps:
|
||
|
||
1. Creates the `kafka` namespace
|
||
2. Installs the Strimzi operator via Helm (scoped to the `kafka` namespace)
|
||
3. Waits for the Strimzi CRDs to be fully established
|
||
4. Applies `kafka.yaml` (KafkaNodePool + Kafka CRs) and waits for `Ready`
|
||
5. Applies `kafka-users.yaml` (KafkaUser CRs) and waits for `Ready`
|
||
6. Applies `kafka-topics.yaml` (the declared topics) and `credentials-access.yaml`
|
||
(the cross-namespace credentials store)
|
||
|
||
Expected duration: **4–6 minutes** on a single-node k3s cluster.
|
||
|
||
### Verify the installation
|
||
|
||
```bash
|
||
kubectl get kafka,kafkanodepool,kafkauser,pods -n kafka
|
||
```
|
||
|
||
Expected output:
|
||
|
||
```
|
||
NAME READY KAFKA VERSION METADATA VERSION
|
||
kafka.kafka.strimzi.io/kafka True 4.1.0 4.1-IV0
|
||
|
||
NAME DESIRED REPLICAS ROLES NODEIDS
|
||
kafkanodepool.kafka.strimzi.io/dual-role 1 ["controller","broker"] [0]
|
||
|
||
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY
|
||
kafkauser.kafka.strimzi.io/kafka-admin kafka scram-sha-512 True
|
||
kafkauser.kafka.strimzi.io/kafka-client kafka scram-sha-512 simple True
|
||
```
|
||
|
||
---
|
||
|
||
## Client configuration
|
||
|
||
### Get the CA certificate
|
||
|
||
```bash
|
||
kubectl -n kafka get secret kafka-cluster-ca-cert \
|
||
-o jsonpath='{.data.ca\.crt}' | base64 -d > kafka-ca.crt
|
||
```
|
||
|
||
### Get user credentials
|
||
|
||
```bash
|
||
# Admin password
|
||
kubectl -n kafka get secret kafka-admin \
|
||
-o jsonpath='{.data.password}' | base64 -d
|
||
|
||
# Client password
|
||
kubectl -n kafka get secret kafka-client \
|
||
-o jsonpath='{.data.password}' | base64 -d
|
||
|
||
# Ready-to-use JAAS config (includes username and password)
|
||
kubectl -n kafka get secret kafka-client \
|
||
-o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d
|
||
```
|
||
|
||
### Sample `client.properties`
|
||
|
||
```properties
|
||
bootstrap.servers=kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
|
||
security.protocol=SASL_SSL
|
||
ssl.truststore.type=PEM
|
||
ssl.truststore.location=/path/to/kafka-ca.crt
|
||
sasl.mechanism=SCRAM-SHA-512
|
||
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
|
||
username="kafka-client" password="<password>";
|
||
```
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
make smoke-test
|
||
```
|
||
|
||
Runs a Kubernetes Job that validates the full produce/consume cycle over SASL_SSL:
|
||
|
||
1. **Topic creation** — creates topic `smoke-test` with `--if-not-exists` (idempotent)
|
||
2. **Produce** — sends 5 messages (`message-1` … `message-5`) via `kafka-console-producer`
|
||
3. **Consume** — reads 5 messages from the beginning via `kafka-console-consumer` and asserts count
|
||
|
||
Expected output:
|
||
```
|
||
==> [1/3] Creating topic 'smoke-test' (idempotent)
|
||
Topic ready.
|
||
==> [2/3] Producing 5 messages
|
||
5 messages produced.
|
||
==> [3/3] Consuming (from-beginning, max 5)
|
||
message-1
|
||
message-2
|
||
message-3
|
||
message-4
|
||
message-5
|
||
==> SMOKE TEST PASSED (5/5 messages)
|
||
==> Smoke test PASSED
|
||
```
|
||
|
||
The job is cleaned up automatically after 1 minute (`ttlSecondsAfterFinished: 60`).
|
||
|
||
---
|
||
|
||
## Adding a new user
|
||
|
||
Create a new `KafkaUser` manifest following the `kafka-client` pattern in `kafka-users.yaml`, then apply it:
|
||
|
||
```bash
|
||
kubectl apply -f kafka-users.yaml
|
||
```
|
||
|
||
Strimzi will create the corresponding secret in the `kafka` namespace within seconds.
|
||
|
||
---
|
||
|
||
## Upgrade
|
||
|
||
> **Rule:** always upgrade the Strimzi operator before upgrading the Kafka version.
|
||
> Check the [Strimzi upgrade guide](https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str) for supported upgrade paths.
|
||
|
||
### Upgrade Kafka version
|
||
|
||
`KAFKA_VER` and `METADATA_VER` are independent because Strimzi recommends upgrading
|
||
the broker binary first, then migrating the metadata format — this allows rollback of
|
||
the binary without being blocked by an already-migrated metadata version.
|
||
|
||
```bash
|
||
# Step 1: upgrade broker binary only
|
||
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV0
|
||
|
||
# Step 2: once confirmed stable, migrate metadata format
|
||
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV1
|
||
```
|
||
|
||
### Upgrade the Strimzi operator
|
||
|
||
```bash
|
||
make upgrade
|
||
```
|
||
|
||
Running `upgrade` without overriding `KAFKA_VER` upgrades only the Strimzi operator
|
||
via `helm upgrade` and leaves the Kafka CR unchanged.
|
||
|
||
---
|
||
|
||
## Uninstallation
|
||
|
||
```bash
|
||
make teardown
|
||
```
|
||
|
||
Deletes in order: KafkaUsers → Kafka cluster → Strimzi operator → namespace.
|
||
|
||
> The `deleteClaim: true` flag in `kafka.yaml` ensures the 10 Gi PVC is deleted
|
||
> together with the KafkaNodePool, leaving no orphaned volumes.
|