infra-kafka/README.md

Kafka on Kubernetes (Strimzi)

Single-broker Kafka cluster for dev/test, deployed via the Strimzi operator on a single-node k3s cluster.

---

Architecture

kafka namespace
├── strimzi-cluster-operator   Strimzi operator — reconciles Kafka/KafkaUser/KafkaTopic CRs
├── kafka-dual-role-0          Kafka 4.3.1 broker (KRaft, controller + broker in one pod)
└── kafka-entity-operator      Topic Operator + User Operator (manages topics and users)

Key design decisions

Choice	Rationale
KRaft mode	No ZooKeeper dependency — simpler, fewer pods
Single node pool dual-role	One pod acts as both controller and broker, suitable for dev/test
SASL_SSL + SCRAM-SHA-512	Authenticated and encrypted connections; TLS managed by Strimzi CA
Simple authorization	ACLs enforced per user; kafka-admin is declared super user
local-path storage (10 Gi)	Default k3s storage class; PVC deleted on cluster teardown

Listeners

Name	Port	Protocol	Auth
tls	9093	SASL_SSL	SCRAM-SHA-512

Bootstrap address (cluster-internal):

kafka-kafka-bootstrap.kafka.svc.cluster.local:9093

Users

KafkaUser	Secret	Role
kafka-admin	kafka-admin	Super user — unrestricted access
kafka-client	kafka-client	Application user — Read/Write all topics and groups

These two are all this repo declares. An application's account is not added here — it belongs to the application, whose own repo says what it may read and write, so that adding or removing the application adds or removes its access. This repo deploys a cluster and holds no opinion about who uses it.

kafka-users.yaml carries the shape to copy. Grant one direction, on one prefix: a producer that can also read every topic exfiltrates every other application's data if it is ever compromised, and a consumer that can write can raise an event someone else acts on.

kafka-client is the wide-open template — useful for a smoke test, and for nothing that runs unattended.

Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:

• password — the SCRAM password
• sasl.jaas.config — the ready-to-use JAAS config string

Topics

This repo declares none, for the same reason it declares no application account: a topic belongs to whoever publishes on it, and its partition count, retention and cleanup policy are that application's decisions, not the broker's.

Declare them as KafkaTopic CRs from the producer's own repo, in this namespace — Strimzi runs with watchAnyNamespace=false, so one created anywhere else is ignored, silently.

Declare them rather than relying on auto-creation: an application account has no Create ACL, so a producer would simply be denied, and an auto-created topic takes broker defaults nobody chose. Partition count is the ceiling on consumer parallelism and cannot be lowered later.

make topic-create TOPIC=… PARTITIONS=… exists for a scratch topic, not for anything an application depends on.

Getting credentials into another namespace

A Secret cannot be mounted across namespaces, and Strimzi writes a KafkaUser's password into this one. credentials-access.yaml therefore declares a ClusterSecretStore named kafka-credentials (External Secrets, kubernetes provider) backed by a ServiceAccount that can read Secrets here and nowhere else. An application namespace then pulls its own password and the cluster CA with an ExternalSecret:

secretStoreRef:
  kind: ClusterSecretStore
  name: kafka-credentials
data:
  - secretKey: password
    remoteRef: {key: <kafkauser-name>, property: password}
  - secretKey: ca.crt
    remoteRef: {key: kafka-cluster-ca-cert, property: ca.crt}

The password stays generated and rotatable by Strimzi; the copy follows. The alternative — copying it by hand — drifts the first time the user is regenerated.

The trade, stated: any namespace allowed to create an ExternalSecret against this store can pull any Kafka account's credentials. Acceptable with one operator; on a shared cluster, narrow the Role with resourceNames and run one store per application.

TLS

Strimzi generates and manages its own internal CA. The cluster CA certificate is stored in:

secret/kafka-cluster-ca-cert  (key: ca.crt)

This certificate must be trusted by any Kafka client connecting over TLS.

---

Prerequisites

• kubectl configured against the target cluster
• helm v3
• make
• cert-manager is not required — Strimzi manages its own CA

---

Makefile

All operations are available via make. Run make help for the full reference.

Cluster lifecycle:
  deploy            Install Strimzi operator + Kafka cluster + users
  smoke-test        Run produce/consume end-to-end test
  upgrade           Upgrade Kafka (KAFKA_VER=) or Strimzi operator
  teardown          Delete the Kafka cluster and namespace

Topics:
  topic-list        List all topics
  topic-create      Create TOPIC (idempotent)
  topic-delete      Delete TOPIC

Messages:
  produce           Produce MESSAGE into TOPIC
  consume           Consume MAX messages from TOPIC (from latest)
  list-messages     Consume MAX messages from TOPIC (from beginning)

Consumer groups:
  group-list        List all consumer groups
  group-offsets     Show offsets and lag for GROUP

Key variables (all overridable on the command line):

Variable	Default	Description
TOPIC	smoke-test	Topic name
MESSAGE	hello	Message to produce
GROUP	my-group	Consumer group
MAX	10	Max messages to consume
PARTITIONS	1	Partitions for topic-create
REPLICAS	1	Replication factor for topic-create
KAFKA_VER	4.1.0	Kafka version for upgrade
METADATA_VER	4.1-IV0	KRaft metadata version for upgrade

---

Installation

make deploy

The target performs the following steps:

1. Creates the kafka namespace
2. Installs the Strimzi operator via Helm (scoped to the kafka namespace)
3. Waits for the Strimzi CRDs to be fully established
4. Applies kafka.yaml (KafkaNodePool + Kafka CRs) and waits for Ready
5. Applies kafka-users.yaml (KafkaUser CRs) and waits for Ready
6. Applies credentials-access.yaml (the cross-namespace credentials store)

Expected duration: 4–6 minutes on a single-node k3s cluster.

Verify the installation

kubectl get kafka,kafkanodepool,kafkauser,pods -n kafka

Expected output:

NAME                           READY   KAFKA VERSION   METADATA VERSION
kafka.kafka.strimzi.io/kafka   True    4.1.0           4.1-IV0

NAME                                       DESIRED REPLICAS   ROLES                     NODEIDS
kafkanodepool.kafka.strimzi.io/dual-role   1                  ["controller","broker"]   [0]

NAME                                      CLUSTER   AUTHENTICATION   AUTHORIZATION   READY
kafkauser.kafka.strimzi.io/kafka-admin    kafka     scram-sha-512                    True
kafkauser.kafka.strimzi.io/kafka-client   kafka     scram-sha-512    simple          True

---

Client configuration

Get the CA certificate

kubectl -n kafka get secret kafka-cluster-ca-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > kafka-ca.crt

Get user credentials

# Admin password
kubectl -n kafka get secret kafka-admin \
  -o jsonpath='{.data.password}' | base64 -d

# Client password
kubectl -n kafka get secret kafka-client \
  -o jsonpath='{.data.password}' | base64 -d

# Ready-to-use JAAS config (includes username and password)
kubectl -n kafka get secret kafka-client \
  -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d

Sample client.properties

bootstrap.servers=kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
security.protocol=SASL_SSL
ssl.truststore.type=PEM
ssl.truststore.location=/path/to/kafka-ca.crt
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="kafka-client" password="<password>";

---

Testing

make smoke-test

Runs a Kubernetes Job that validates the full produce/consume cycle over SASL_SSL:

1. Topic creation — creates topic smoke-test with --if-not-exists (idempotent)
2. Produce — sends 5 messages (message-1 … message-5) via kafka-console-producer
3. Consume — reads 5 messages from the beginning via kafka-console-consumer and asserts count

Expected output:

==> [1/3] Creating topic 'smoke-test' (idempotent)
    Topic ready.
==> [2/3] Producing 5 messages
    5 messages produced.
==> [3/3] Consuming (from-beginning, max 5)
message-1
message-2
message-3
message-4
message-5
==> SMOKE TEST PASSED (5/5 messages)
==> Smoke test PASSED

The job is cleaned up automatically after 1 minute (ttlSecondsAfterFinished: 60).

---

Adding a new user

Copy the pattern documented at the bottom of kafka-users.yaml into the application's own repo, then apply it — into this namespace, since Strimzi watches no other:

kubectl apply -f kafka-users.yaml

Strimzi will create the corresponding secret in the kafka namespace within seconds.

---

Upgrade

Rule: always upgrade the Strimzi operator before upgrading the Kafka version. Check the Strimzi upgrade guide for supported upgrade paths.

Upgrade Kafka version

KAFKA_VER and METADATA_VER are independent because Strimzi recommends upgrading the broker binary first, then migrating the metadata format — this allows rollback of the binary without being blocked by an already-migrated metadata version.

# Step 1: upgrade broker binary only
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV0

# Step 2: once confirmed stable, migrate metadata format
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV1

Upgrade the Strimzi operator

make upgrade

Running upgrade without overriding KAFKA_VER upgrades only the Strimzi operator via helm upgrade and leaves the Kafka CR unchanged.

---

Uninstallation

make teardown

Deletes in order: KafkaUsers → Kafka cluster → Strimzi operator → namespace.

The deleteClaim: true flag in kafka.yaml ensures the 10 Gi PVC is deleted together with the KafkaNodePool, leaving no orphaned volumes.