Stéphane Tailland
e6699912a5
The repo had accreted one estate's accounts, topics and prose. An application's account and its topics belong to the application, whose own repo says what it may read and write — so that adding or removing the application adds or removes its access. What is left is the cluster, the admin and smoke-test accounts, the cross-namespace credentials store, and the pattern to copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
333 lines
10 KiB
Markdown
333 lines
10 KiB
Markdown
# Kafka on Kubernetes (Strimzi)
|
||
|
||
Single-broker Kafka cluster for dev/test, deployed via the [Strimzi](https://strimzi.io/) operator on a single-node k3s cluster.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
kafka namespace
|
||
├── strimzi-cluster-operator Strimzi operator — reconciles Kafka/KafkaUser/KafkaTopic CRs
|
||
├── kafka-dual-role-0 Kafka 4.3.1 broker (KRaft, controller + broker in one pod)
|
||
└── kafka-entity-operator Topic Operator + User Operator (manages topics and users)
|
||
```
|
||
|
||
### Key design decisions
|
||
|
||
| Choice | Rationale |
|
||
|---|---|
|
||
| **KRaft mode** | No ZooKeeper dependency — simpler, fewer pods |
|
||
| **Single node pool** `dual-role` | One pod acts as both controller and broker, suitable for dev/test |
|
||
| **SASL_SSL + SCRAM-SHA-512** | Authenticated and encrypted connections; TLS managed by Strimzi CA |
|
||
| **Simple authorization** | ACLs enforced per user; `kafka-admin` is declared super user |
|
||
| **`local-path` storage (10 Gi)** | Default k3s storage class; PVC deleted on cluster teardown |
|
||
|
||
### Listeners
|
||
|
||
| Name | Port | Protocol | Auth |
|
||
|---|---|---|---|
|
||
| `tls` | 9093 | SASL_SSL | SCRAM-SHA-512 |
|
||
|
||
Bootstrap address (cluster-internal):
|
||
```
|
||
kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
|
||
```
|
||
|
||
### Users
|
||
|
||
| KafkaUser | Secret | Role |
|
||
|---|---|---|
|
||
| `kafka-admin` | `kafka-admin` | Super user — unrestricted access |
|
||
| `kafka-client` | `kafka-client` | Application user — Read/Write all topics and groups |
|
||
|
||
These two are all this repo declares. **An application's account is not added
|
||
here** — it belongs to the application, whose own repo says what it may read and
|
||
write, so that adding or removing the application adds or removes its access.
|
||
This repo deploys a cluster and holds no opinion about who uses it.
|
||
|
||
`kafka-users.yaml` carries the shape to copy. Grant **one direction, on one
|
||
prefix**: a producer that can also read every topic exfiltrates every other
|
||
application's data if it is ever compromised, and a consumer that can write can
|
||
raise an event someone else acts on.
|
||
|
||
`kafka-client` is the wide-open template — useful for a smoke test, and for
|
||
nothing that runs unattended.
|
||
|
||
Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:
|
||
- `password` — the SCRAM password
|
||
- `sasl.jaas.config` — the ready-to-use JAAS config string
|
||
|
||
### Topics
|
||
|
||
This repo declares none, for the same reason it declares no application account:
|
||
a topic belongs to whoever publishes on it, and its partition count, retention
|
||
and cleanup policy are that application's decisions, not the broker's.
|
||
|
||
Declare them as `KafkaTopic` CRs from the producer's own repo, **in this
|
||
namespace** — Strimzi runs with `watchAnyNamespace=false`, so one created
|
||
anywhere else is ignored, silently.
|
||
|
||
Declare them rather than relying on auto-creation: an application account has no
|
||
`Create` ACL, so a producer would simply be denied, and an auto-created topic
|
||
takes broker defaults nobody chose. Partition count is the ceiling on consumer
|
||
parallelism and **cannot be lowered later**.
|
||
|
||
`make topic-create TOPIC=… PARTITIONS=…` exists for a scratch topic, not for
|
||
anything an application depends on.
|
||
|
||
### Getting credentials into another namespace
|
||
|
||
A Secret cannot be mounted across namespaces, and Strimzi writes a KafkaUser's
|
||
password into **this** one. `credentials-access.yaml` therefore declares a
|
||
`ClusterSecretStore` named `kafka-credentials` (External Secrets, `kubernetes`
|
||
provider) backed by a ServiceAccount that can read Secrets here and nowhere
|
||
else. An application namespace then pulls its own password and the cluster CA
|
||
with an `ExternalSecret`:
|
||
|
||
```yaml
|
||
secretStoreRef:
|
||
kind: ClusterSecretStore
|
||
name: kafka-credentials
|
||
data:
|
||
- secretKey: password
|
||
remoteRef: {key: <kafkauser-name>, property: password}
|
||
- secretKey: ca.crt
|
||
remoteRef: {key: kafka-cluster-ca-cert, property: ca.crt}
|
||
```
|
||
|
||
The password stays generated and rotatable by Strimzi; the copy follows. The
|
||
alternative — copying it by hand — drifts the first time the user is
|
||
regenerated.
|
||
|
||
**The trade, stated:** any namespace allowed to create an `ExternalSecret`
|
||
against this store can pull **any** Kafka account's credentials. Acceptable with
|
||
one operator; on a shared cluster, narrow the Role with `resourceNames` and run
|
||
one store per application.
|
||
|
||
### TLS
|
||
|
||
Strimzi generates and manages its own internal CA. The cluster CA certificate is stored in:
|
||
```
|
||
secret/kafka-cluster-ca-cert (key: ca.crt)
|
||
```
|
||
|
||
This certificate must be trusted by any Kafka client connecting over TLS.
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- `kubectl` configured against the target cluster
|
||
- `helm` v3
|
||
- `make`
|
||
- `cert-manager` is **not** required — Strimzi manages its own CA
|
||
|
||
---
|
||
|
||
## Makefile
|
||
|
||
All operations are available via `make`. Run `make help` for the full reference.
|
||
|
||
```
|
||
Cluster lifecycle:
|
||
deploy Install Strimzi operator + Kafka cluster + users
|
||
smoke-test Run produce/consume end-to-end test
|
||
upgrade Upgrade Kafka (KAFKA_VER=) or Strimzi operator
|
||
teardown Delete the Kafka cluster and namespace
|
||
|
||
Topics:
|
||
topic-list List all topics
|
||
topic-create Create TOPIC (idempotent)
|
||
topic-delete Delete TOPIC
|
||
|
||
Messages:
|
||
produce Produce MESSAGE into TOPIC
|
||
consume Consume MAX messages from TOPIC (from latest)
|
||
list-messages Consume MAX messages from TOPIC (from beginning)
|
||
|
||
Consumer groups:
|
||
group-list List all consumer groups
|
||
group-offsets Show offsets and lag for GROUP
|
||
```
|
||
|
||
Key variables (all overridable on the command line):
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `TOPIC` | `smoke-test` | Topic name |
|
||
| `MESSAGE` | `hello` | Message to produce |
|
||
| `GROUP` | `my-group` | Consumer group |
|
||
| `MAX` | `10` | Max messages to consume |
|
||
| `PARTITIONS` | `1` | Partitions for topic-create |
|
||
| `REPLICAS` | `1` | Replication factor for topic-create |
|
||
| `KAFKA_VER` | `4.1.0` | Kafka version for upgrade |
|
||
| `METADATA_VER` | `4.1-IV0` | KRaft metadata version for upgrade |
|
||
|
||
---
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
make deploy
|
||
```
|
||
|
||
The target performs the following steps:
|
||
|
||
1. Creates the `kafka` namespace
|
||
2. Installs the Strimzi operator via Helm (scoped to the `kafka` namespace)
|
||
3. Waits for the Strimzi CRDs to be fully established
|
||
4. Applies `kafka.yaml` (KafkaNodePool + Kafka CRs) and waits for `Ready`
|
||
5. Applies `kafka-users.yaml` (KafkaUser CRs) and waits for `Ready`
|
||
6. Applies `credentials-access.yaml` (the cross-namespace credentials store)
|
||
|
||
Expected duration: **4–6 minutes** on a single-node k3s cluster.
|
||
|
||
### Verify the installation
|
||
|
||
```bash
|
||
kubectl get kafka,kafkanodepool,kafkauser,pods -n kafka
|
||
```
|
||
|
||
Expected output:
|
||
|
||
```
|
||
NAME READY KAFKA VERSION METADATA VERSION
|
||
kafka.kafka.strimzi.io/kafka True 4.1.0 4.1-IV0
|
||
|
||
NAME DESIRED REPLICAS ROLES NODEIDS
|
||
kafkanodepool.kafka.strimzi.io/dual-role 1 ["controller","broker"] [0]
|
||
|
||
NAME CLUSTER AUTHENTICATION AUTHORIZATION READY
|
||
kafkauser.kafka.strimzi.io/kafka-admin kafka scram-sha-512 True
|
||
kafkauser.kafka.strimzi.io/kafka-client kafka scram-sha-512 simple True
|
||
```
|
||
|
||
---
|
||
|
||
## Client configuration
|
||
|
||
### Get the CA certificate
|
||
|
||
```bash
|
||
kubectl -n kafka get secret kafka-cluster-ca-cert \
|
||
-o jsonpath='{.data.ca\.crt}' | base64 -d > kafka-ca.crt
|
||
```
|
||
|
||
### Get user credentials
|
||
|
||
```bash
|
||
# Admin password
|
||
kubectl -n kafka get secret kafka-admin \
|
||
-o jsonpath='{.data.password}' | base64 -d
|
||
|
||
# Client password
|
||
kubectl -n kafka get secret kafka-client \
|
||
-o jsonpath='{.data.password}' | base64 -d
|
||
|
||
# Ready-to-use JAAS config (includes username and password)
|
||
kubectl -n kafka get secret kafka-client \
|
||
-o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d
|
||
```
|
||
|
||
### Sample `client.properties`
|
||
|
||
```properties
|
||
bootstrap.servers=kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
|
||
security.protocol=SASL_SSL
|
||
ssl.truststore.type=PEM
|
||
ssl.truststore.location=/path/to/kafka-ca.crt
|
||
sasl.mechanism=SCRAM-SHA-512
|
||
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
|
||
username="kafka-client" password="<password>";
|
||
```
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
make smoke-test
|
||
```
|
||
|
||
Runs a Kubernetes Job that validates the full produce/consume cycle over SASL_SSL:
|
||
|
||
1. **Topic creation** — creates topic `smoke-test` with `--if-not-exists` (idempotent)
|
||
2. **Produce** — sends 5 messages (`message-1` … `message-5`) via `kafka-console-producer`
|
||
3. **Consume** — reads 5 messages from the beginning via `kafka-console-consumer` and asserts count
|
||
|
||
Expected output:
|
||
```
|
||
==> [1/3] Creating topic 'smoke-test' (idempotent)
|
||
Topic ready.
|
||
==> [2/3] Producing 5 messages
|
||
5 messages produced.
|
||
==> [3/3] Consuming (from-beginning, max 5)
|
||
message-1
|
||
message-2
|
||
message-3
|
||
message-4
|
||
message-5
|
||
==> SMOKE TEST PASSED (5/5 messages)
|
||
==> Smoke test PASSED
|
||
```
|
||
|
||
The job is cleaned up automatically after 1 minute (`ttlSecondsAfterFinished: 60`).
|
||
|
||
---
|
||
|
||
## Adding a new user
|
||
|
||
Copy the pattern documented at the bottom of `kafka-users.yaml` into the
|
||
application's **own** repo, then apply it — into this namespace, since Strimzi
|
||
watches no other:
|
||
|
||
```bash
|
||
kubectl apply -f kafka-users.yaml
|
||
```
|
||
|
||
Strimzi will create the corresponding secret in the `kafka` namespace within seconds.
|
||
|
||
---
|
||
|
||
## Upgrade
|
||
|
||
> **Rule:** always upgrade the Strimzi operator before upgrading the Kafka version.
|
||
> Check the [Strimzi upgrade guide](https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str) for supported upgrade paths.
|
||
|
||
### Upgrade Kafka version
|
||
|
||
`KAFKA_VER` and `METADATA_VER` are independent because Strimzi recommends upgrading
|
||
the broker binary first, then migrating the metadata format — this allows rollback of
|
||
the binary without being blocked by an already-migrated metadata version.
|
||
|
||
```bash
|
||
# Step 1: upgrade broker binary only
|
||
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV0
|
||
|
||
# Step 2: once confirmed stable, migrate metadata format
|
||
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV1
|
||
```
|
||
|
||
### Upgrade the Strimzi operator
|
||
|
||
```bash
|
||
make upgrade
|
||
```
|
||
|
||
Running `upgrade` without overriding `KAFKA_VER` upgrades only the Strimzi operator
|
||
via `helm upgrade` and leaves the Kafka CR unchanged.
|
||
|
||
---
|
||
|
||
## Uninstallation
|
||
|
||
```bash
|
||
make teardown
|
||
```
|
||
|
||
Deletes in order: KafkaUsers → Kafka cluster → Strimzi operator → namespace.
|
||
|
||
> The `deleteClaim: true` flag in `kafka.yaml` ensures the 10 Gi PVC is deleted
|
||
> together with the KafkaNodePool, leaving no orphaned volumes.
|