infra-kafka/README.md

Kafka on Kubernetes (Strimzi)

Single-broker Kafka cluster for dev/test, deployed via the Strimzi operator on a single-node k3s cluster.

---

Architecture

kafka namespace
├── strimzi-cluster-operator   Strimzi operator — reconciles Kafka/KafkaUser/KafkaTopic CRs
├── kafka-dual-role-0          Kafka 4.1.0 broker (KRaft, controller + broker in one pod)
└── kafka-entity-operator      Topic Operator + User Operator (manages topics and users)

Key design decisions

Choice	Rationale
KRaft mode	No ZooKeeper dependency — simpler, fewer pods
Single node pool dual-role	One pod acts as both controller and broker, suitable for dev/test
SASL_SSL + SCRAM-SHA-512	Authenticated and encrypted connections; TLS managed by Strimzi CA
Simple authorization	ACLs enforced per user; kafka-admin is declared super user
local-path storage (10 Gi)	Default k3s storage class; PVC deleted on cluster teardown

Listeners

Name	Port	Protocol	Auth
tls	9093	SASL_SSL	SCRAM-SHA-512

Bootstrap address (cluster-internal):

kafka-kafka-bootstrap.kafka.svc.cluster.local:9093

Users

KafkaUser	Secret	Role
kafka-admin	kafka-admin	Super user — unrestricted access
kafka-client	kafka-client	Application user — Read/Write all topics and groups

Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:

• password — the SCRAM password
• sasl.jaas.config — the ready-to-use JAAS config string

TLS

Strimzi generates and manages its own internal CA. The cluster CA certificate is stored in:

secret/kafka-cluster-ca-cert  (key: ca.crt)

This certificate must be trusted by any Kafka client connecting over TLS.

---

Prerequisites

• kubectl configured against the target cluster
• helm v3
• make
• cert-manager is not required — Strimzi manages its own CA

---

Makefile

All operations are available via make. Run make help for the full reference.

Cluster lifecycle:
  deploy            Install Strimzi operator + Kafka cluster + users
  smoke-test        Run produce/consume end-to-end test
  upgrade           Upgrade Kafka (KAFKA_VER=) or Strimzi operator
  teardown          Delete the Kafka cluster and namespace

Topics:
  topic-list        List all topics
  topic-create      Create TOPIC (idempotent)
  topic-delete      Delete TOPIC

Messages:
  produce           Produce MESSAGE into TOPIC
  consume           Consume MAX messages from TOPIC (from latest)
  list-messages     Consume MAX messages from TOPIC (from beginning)

Consumer groups:
  group-list        List all consumer groups
  group-offsets     Show offsets and lag for GROUP

Key variables (all overridable on the command line):

Variable	Default	Description
TOPIC	smoke-test	Topic name
MESSAGE	hello	Message to produce
GROUP	my-group	Consumer group
MAX	10	Max messages to consume
PARTITIONS	1	Partitions for topic-create
REPLICAS	1	Replication factor for topic-create
KAFKA_VER	4.1.0	Kafka version for upgrade
METADATA_VER	4.1-IV0	KRaft metadata version for upgrade

---

Installation

make deploy

The target performs the following steps:

1. Creates the kafka namespace
2. Installs the Strimzi operator via Helm (scoped to the kafka namespace)
3. Waits for the Strimzi CRDs to be fully established
4. Applies kafka.yaml (KafkaNodePool + Kafka CRs) and waits for Ready
5. Applies kafka-users.yaml (KafkaUser CRs) and waits for Ready

Expected duration: 4–6 minutes on a single-node k3s cluster.

Verify the installation

kubectl get kafka,kafkanodepool,kafkauser,pods -n kafka

Expected output:

NAME                           READY   KAFKA VERSION   METADATA VERSION
kafka.kafka.strimzi.io/kafka   True    4.1.0           4.1-IV0

NAME                                       DESIRED REPLICAS   ROLES                     NODEIDS
kafkanodepool.kafka.strimzi.io/dual-role   1                  ["controller","broker"]   [0]

NAME                                      CLUSTER   AUTHENTICATION   AUTHORIZATION   READY
kafkauser.kafka.strimzi.io/kafka-admin    kafka     scram-sha-512                    True
kafkauser.kafka.strimzi.io/kafka-client   kafka     scram-sha-512    simple          True

---

Client configuration

Get the CA certificate

kubectl -n kafka get secret kafka-cluster-ca-cert \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > kafka-ca.crt

Get user credentials

# Admin password
kubectl -n kafka get secret kafka-admin \
  -o jsonpath='{.data.password}' | base64 -d

# Client password
kubectl -n kafka get secret kafka-client \
  -o jsonpath='{.data.password}' | base64 -d

# Ready-to-use JAAS config (includes username and password)
kubectl -n kafka get secret kafka-client \
  -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d

Sample client.properties

bootstrap.servers=kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
security.protocol=SASL_SSL
ssl.truststore.type=PEM
ssl.truststore.location=/path/to/kafka-ca.crt
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="kafka-client" password="<password>";

---

Testing

make smoke-test

Runs a Kubernetes Job that validates the full produce/consume cycle over SASL_SSL:

1. Topic creation — creates topic smoke-test with --if-not-exists (idempotent)
2. Produce — sends 5 messages (message-1 … message-5) via kafka-console-producer
3. Consume — reads 5 messages from the beginning via kafka-console-consumer and asserts count

Expected output:

==> [1/3] Creating topic 'smoke-test' (idempotent)
    Topic ready.
==> [2/3] Producing 5 messages
    5 messages produced.
==> [3/3] Consuming (from-beginning, max 5)
message-1
message-2
message-3
message-4
message-5
==> SMOKE TEST PASSED (5/5 messages)
==> Smoke test PASSED

The job is cleaned up automatically after 1 minute (ttlSecondsAfterFinished: 60).

---

Adding a new user

Create a new KafkaUser manifest following the kafka-client pattern in kafka-users.yaml, then apply it:

kubectl apply -f kafka-users.yaml

Strimzi will create the corresponding secret in the kafka namespace within seconds.

---

Upgrade

Rule: always upgrade the Strimzi operator before upgrading the Kafka version. Check the Strimzi upgrade guide for supported upgrade paths.

Upgrade Kafka version

KAFKA_VER and METADATA_VER are independent because Strimzi recommends upgrading the broker binary first, then migrating the metadata format — this allows rollback of the binary without being blocked by an already-migrated metadata version.

# Step 1: upgrade broker binary only
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV0

# Step 2: once confirmed stable, migrate metadata format
make upgrade KAFKA_VER=4.1.1 METADATA_VER=4.1-IV1

Upgrade the Strimzi operator

make upgrade

Running upgrade without overriding KAFKA_VER upgrades only the Strimzi operator via helm upgrade and leaves the Kafka CR unchanged.

---

Uninstallation

make teardown

Deletes in order: KafkaUsers → Kafka cluster → Strimzi operator → namespace.

The deleteClaim: true flag in kafka.yaml ensures the 10 Gi PVC is deleted together with the KafkaNodePool, leaving no orphaned volumes.