# Kafka on Kubernetes (Strimzi) Single-broker Kafka cluster for dev/test, deployed via the [Strimzi](https://strimzi.io/) operator on a single-node k3s cluster. --- ## Architecture ``` kafka namespace ├── strimzi-cluster-operator Strimzi operator — reconciles Kafka/KafkaUser/KafkaTopic CRs ├── kafka-dual-role-0 Kafka 4.3.1 broker (KRaft, controller + broker in one pod) └── kafka-entity-operator Topic Operator + User Operator (manages topics and users) ``` ### Key design decisions | Choice | Rationale | |---|---| | **KRaft mode** | No ZooKeeper dependency — simpler, fewer pods | | **Single node pool** `dual-role` | One pod acts as both controller and broker, suitable for dev/test | | **SASL_SSL + SCRAM-SHA-512** | Authenticated and encrypted connections; TLS managed by Strimzi CA | | **Simple authorization** | ACLs enforced per user; `kafka-admin` is declared super user | | **`local-path` storage (10 Gi)** | Default k3s storage class; PVC deleted on cluster teardown | ### Listeners | Name | Port | Protocol | Auth | |---|---|---|---| | `tls` | 9093 | SASL_SSL | SCRAM-SHA-512 | Bootstrap address (cluster-internal): ``` kafka-kafka-bootstrap.kafka.svc.cluster.local:9093 ``` ### Users | KafkaUser | Secret | Role | |---|---|---| | `kafka-admin` | `kafka-admin` | Super user — unrestricted access | | `kafka-client` | `kafka-client` | Application user — Read/Write all topics and groups | These two are all this repo declares. **An application's account is not added here** — it belongs to the application, whose own repo says what it may read and write, so that adding or removing the application adds or removes its access. This repo deploys a cluster and holds no opinion about who uses it. `kafka-users.yaml` carries the shape to copy. Grant **one direction, on one prefix**: a producer that can also read every topic exfiltrates every other application's data if it is ever compromised, and a consumer that can write can raise an event someone else acts on. `kafka-client` is the wide-open template — useful for a smoke test, and for nothing that runs unattended. Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys: - `password` — the SCRAM password - `sasl.jaas.config` — the ready-to-use JAAS config string ### Topics This repo declares none, for the same reason it declares no application account: a topic belongs to whoever publishes on it, and its partition count, retention and cleanup policy are that application's decisions, not the broker's. Declare them as `KafkaTopic` CRs from the producer's own repo, **in this namespace** — Strimzi runs with `watchAnyNamespace=false`, so one created anywhere else is ignored, silently. Declare them rather than relying on auto-creation: an application account has no `Create` ACL, so a producer would simply be denied, and an auto-created topic takes broker defaults nobody chose. Partition count is the ceiling on consumer parallelism and **cannot be lowered later**. `make topic-create TOPIC=… PARTITIONS=…` exists for a scratch topic, not for anything an application depends on. ### Getting credentials into another namespace A Secret cannot be mounted across namespaces, and Strimzi writes a KafkaUser's password into **this** one. `credentials-access.yaml` therefore declares a `ClusterSecretStore` named `kafka-credentials` (External Secrets, `kubernetes` provider) backed by a ServiceAccount that can read Secrets here and nowhere else. An application namespace then pulls its own password and the cluster CA with an `ExternalSecret`: ```yaml secretStoreRef: kind: ClusterSecretStore name: kafka-credentials data: - secretKey: password remoteRef: {key: , property: password} - secretKey: ca.crt remoteRef: {key: kafka-cluster-ca-cert, property: ca.crt} ``` The password stays generated and rotatable by Strimzi; the copy follows. The alternative — copying it by hand — drifts the first time the user is regenerated. **The trade, stated:** any namespace allowed to create an `ExternalSecret` against this store can pull **any** Kafka account's credentials. Acceptable with one operator; on a shared cluster, narrow the Role with `resourceNames` and run one store per application. ### TLS Strimzi generates and manages its own internal CA. The cluster CA certificate is stored in: ``` secret/kafka-cluster-ca-cert (key: ca.crt) ``` This certificate must be trusted by any Kafka client connecting over TLS. --- ## Prerequisites - `kubectl` configured against the target cluster - `helm` v3 - `make` - `cert-manager` is **not** required — Strimzi manages its own CA --- ## Makefile All operations are available via `make`. Run `make help` for the full reference. ``` Cluster lifecycle: deploy Install Strimzi operator + Kafka cluster + users smoke-test Run produce/consume end-to-end test upgrade Upgrade Kafka (KAFKA_VER=) or Strimzi operator teardown Delete the Kafka cluster and namespace Topics: topic-list List all topics topic-create Create TOPIC (idempotent) topic-delete Delete TOPIC Messages: produce Produce MESSAGE into TOPIC consume Consume MAX messages from TOPIC (from latest) list-messages Consume MAX messages from TOPIC (from beginning) Consumer groups: group-list List all consumer groups group-offsets Show offsets and lag for GROUP ``` Key variables (all overridable on the command line): | Variable | Default | Description | |---|---|---| | `TOPIC` | `smoke-test` | Topic name | | `MESSAGE` | `hello` | Message to produce | | `GROUP` | `my-group` | Consumer group | | `MAX` | `10` | Max messages to consume | | `PARTITIONS` | `1` | Partitions for topic-create | | `REPLICAS` | `1` | Replication factor for topic-create | | `KAFKA_VER` | `4.1.0` | Kafka version for upgrade | | `METADATA_VER` | `4.1-IV0` | KRaft metadata version for upgrade | --- ## Installation ```bash make deploy ``` The target performs the following steps: 1. Creates the `kafka` namespace 2. Installs the Strimzi operator via Helm (scoped to the `kafka` namespace) 3. Waits for the Strimzi CRDs to be fully established 4. Applies `kafka.yaml` (KafkaNodePool + Kafka CRs) and waits for `Ready` 5. Applies `kafka-users.yaml` (KafkaUser CRs) and waits for `Ready` 6. Applies `credentials-access.yaml` (the cross-namespace credentials store) Expected duration: **4–6 minutes** on a single-node k3s cluster. ### Verify the installation ```bash kubectl get kafka,kafkanodepool,kafkauser,pods -n kafka ``` Expected output: ``` NAME READY KAFKA VERSION METADATA VERSION kafka.kafka.strimzi.io/kafka True 4.1.0 4.1-IV0 NAME DESIRED REPLICAS ROLES NODEIDS kafkanodepool.kafka.strimzi.io/dual-role 1 ["controller","broker"] [0] NAME CLUSTER AUTHENTICATION AUTHORIZATION READY kafkauser.kafka.strimzi.io/kafka-admin kafka scram-sha-512 True kafkauser.kafka.strimzi.io/kafka-client kafka scram-sha-512 simple True ``` --- ## Client configuration ### Get the CA certificate ```bash kubectl -n kafka get secret kafka-cluster-ca-cert \ -o jsonpath='{.data.ca\.crt}' | base64 -d > kafka-ca.crt ``` ### Get user credentials ```bash # Admin password kubectl -n kafka get secret kafka-admin \ -o jsonpath='{.data.password}' | base64 -d # Client password kubectl -n kafka get secret kafka-client \ -o jsonpath='{.data.password}' | base64 -d # Ready-to-use JAAS config (includes username and password) kubectl -n kafka get secret kafka-client \ -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d ``` ### Sample `client.properties` ```properties bootstrap.servers=kafka-kafka-bootstrap.kafka.svc.cluster.local:9093 security.protocol=SASL_SSL ssl.truststore.type=PEM ssl.truststore.location=/path/to/kafka-ca.crt sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="kafka-client" password=""; ``` --- ## Testing ```bash make smoke-test ``` Runs a Kubernetes Job that validates the full produce/consume cycle over SASL_SSL: 1. **Topic creation** — creates topic `smoke-test` with `--if-not-exists` (idempotent) 2. **Produce** — sends 5 messages (`message-1` … `message-5`) via `kafka-console-producer` 3. **Consume** — reads 5 messages from the beginning via `kafka-console-consumer` and asserts count Expected output: ``` ==> [1/3] Creating topic 'smoke-test' (idempotent) Topic ready. ==> [2/3] Producing 5 messages 5 messages produced. ==> [3/3] Consuming (from-beginning, max 5) message-1 message-2 message-3 message-4 message-5 ==> SMOKE TEST PASSED (5/5 messages) ==> Smoke test PASSED ``` The job is cleaned up automatically after 1 minute (`ttlSecondsAfterFinished: 60`). --- ## Adding a new user Copy the pattern documented at the bottom of `kafka-users.yaml` into the application's **own** repo, then apply it — into this namespace, since Strimzi watches no other: ```bash kubectl apply -f kafka-users.yaml ``` Strimzi will create the corresponding secret in the `kafka` namespace within seconds. --- ## Upgrade > **Rule:** always upgrade the Strimzi operator before upgrading the Kafka version. > Check the [Strimzi upgrade guide](https://strimzi.io/docs/operators/latest/deploying.html#assembly-upgrade-str) for supported upgrade paths. ### Upgrade Kafka version `KAFKA_VER` and `METADATA_VER` are independent because Strimzi recommends upgrading the broker binary first, then migrating the metadata format — this allows rollback of the binary without being blocked by an already-migrated metadata version. ```bash # Step 1: upgrade broker binary only make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV0 # Step 2: once confirmed stable, migrate metadata format make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV1 ``` ### Upgrade the Strimzi operator ```bash make upgrade ``` Running `upgrade` without overriding `KAFKA_VER` upgrades only the Strimzi operator via `helm upgrade` and leaves the Kafka CR unchanged. --- ## Uninstallation ```bash make teardown ``` Deletes in order: KafkaUsers → Kafka cluster → Strimzi operator → namespace. > The `deleteClaim: true` flag in `kafka.yaml` ensures the 10 Gi PVC is deleted > together with the KafkaNodePool, leaving no orphaned volumes.