stt/infra-kafka
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deploy a cluster, not an estate

Browse Source 
The repo had accreted one estate's accounts, topics and prose. An application's
account and its topics belong to the application, whose own repo says what it
may read and write — so that adding or removing the application adds or removes
its access.

What is left is the cluster, the admin and smoke-test accounts, the
cross-namespace credentials store, and the pattern to copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Stéphane Tailland
2026-08-31 08:43:14 +02:00
parent 979c9da6c0
commit e6699912a5
4 changed files with 62 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Makefile
+1 -3
View File
@@ -81,8 +81,7 @@ deploy:
@for user in kafka-admin kafka-client; do \
kubectl wait kafkauser/$$user --for=condition=Ready --timeout=120s -n $(NAMESPACE); \
done
@echo "==> [6/6] Deploying topics and the credentials store"
@kubectl apply -f $(SCRIPT_DIR)kafka-topics.yaml
@echo "==> [6/6] Deploying the credentials store"
@kubectl apply -f $(SCRIPT_DIR)credentials-access.yaml
@echo ""
@echo "Cluster ready. Bootstrap: $(BOOTSTRAP)"
@@ -116,7 +115,6 @@ upgrade:
teardown:
@echo "==> Deleting KafkaUsers and Kafka cluster"
@kubectl delete -f $(SCRIPT_DIR)credentials-access.yaml --ignore-not-found
@kubectl delete -f $(SCRIPT_DIR)kafka-topics.yaml --ignore-not-found
@kubectl delete -f $(SCRIPT_DIR)kafka-users.yaml --ignore-not-found
@kubectl delete -f $(SCRIPT_DIR)kafka.yaml --ignore-not-found
@echo "==> Waiting for pods to terminate"
README.md
+27 -26
View File
@@ -40,18 +40,19 @@ kafka-kafka-bootstrap.kafka.svc.cluster.local:9093
|---|---|---|
| `kafka-admin` | `kafka-admin` | Super user — unrestricted access |
| `kafka-client` | `kafka-client` | Application user — Read/Write all topics and groups |
| `classic-models-api` | `classic-models-api` | Producer for `classic-models.*` — **Write and Describe only**, no Read |
| `classic-models-agent-approvisionnement` | same name | Consumer of `classic-models.catalog` and group prefix `approvisionnement-` — **Read and Describe only**, no Write |
Each application account is scoped to what it actually does. The producer
publishes domain events and consumes nothing, so it is write-only: one that
could also read every topic exfiltrates every other domain's events if it is
ever compromised. The consumer is the mirror — it cannot see sales or payments,
and it cannot write, so a compromised workflow can raise no event that another
agent would then act on.
These two are all this repo declares. **An application's account is not added
here** — it belongs to the application, whose own repo says what it may read and
write, so that adding or removing the application adds or removes its access.
This repo deploys a cluster and holds no opinion about who uses it.
Duplicate one of these per application. `kafka-client` is the wide-open
template, useful for a smoke test and for nothing that runs unattended.
`kafka-users.yaml` carries the shape to copy. Grant **one direction, on one
prefix**: a producer that can also read every topic exfiltrates every other
application's data if it is ever compromised, and a consumer that can write can
raise an event someone else acts on.
`kafka-client` is the wide-open template — useful for a smoke test, and for
nothing that runs unattended.
Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:
- `password` — the SCRAM password
@@ -59,22 +60,21 @@ Strimzi stores credentials in Kubernetes secrets. Each secret contains two keys:
### Topics
Declared in `kafka-topics.yaml`, never auto-created — an application account has
no `Create` ACL, so a producer would simply be denied, and an auto-created topic
takes broker defaults nobody chose.
This repo declares none, for the same reason it declares no application account:
a topic belongs to whoever publishes on it, and its partition count, retention
and cleanup policy are that application's decisions, not the broker's.
| Topic | Partitions | Replicas | Retention |
|---|---|---|---|
| `classic-models.catalog` | 3 | 1 | 7 days |
| `classic-models.sales` | 3 | 1 | 7 days |
| `classic-models.payments` | 3 | 1 | 7 days |
Declare them as `KafkaTopic` CRs from the producer's own repo, **in this
namespace** — Strimzi runs with `watchAnyNamespace=false`, so one created
anywhere else is ignored, silently.
Partition count is the ceiling on consumer parallelism and **cannot be lowered
later**, so it is set for the traffic expected rather than the traffic today.
Declare them rather than relying on auto-creation: an application account has no
`Create` ACL, so a producer would simply be denied, and an auto-created topic
takes broker defaults nobody chose. Partition count is the ceiling on consumer
parallelism and **cannot be lowered later**.
The events these carry are specified by their producer, in
`classic-models-api/docs/DOMAIN_EVENTS.md`. This repo owns the cluster, not the
contract.
`make topic-create TOPIC=… PARTITIONS=…` exists for a scratch topic, not for
anything an application depends on.
### Getting credentials into another namespace
@@ -179,8 +179,7 @@ The target performs the following steps:
3. Waits for the Strimzi CRDs to be fully established
4. Applies `kafka.yaml` (KafkaNodePool + Kafka CRs) and waits for `Ready`
5. Applies `kafka-users.yaml` (KafkaUser CRs) and waits for `Ready`
6. Applies `kafka-topics.yaml` (the declared topics) and `credentials-access.yaml`
(the cross-namespace credentials store)
6. Applies `credentials-access.yaml` (the cross-namespace credentials store)
Expected duration: **4–6 minutes** on a single-node k3s cluster.
@@ -279,7 +278,9 @@ The job is cleaned up automatically after 1 minute (`ttlSecondsAfterFinished: 60
## Adding a new user
Create a new `KafkaUser` manifest following the `kafka-client` pattern in `kafka-users.yaml`, then apply it:
Copy the pattern documented at the bottom of `kafka-users.yaml` into the
application's **own** repo, then apply it — into this namespace, since Strimzi
watches no other:
```bash
kubectl apply -f kafka-users.yaml
kafka-topics.yaml
-56
View File
@@ -1,56 +0,0 @@
---
# Topics are declared, never auto-created.
#
# Two reasons. An application account has no Create ACL — a producer that can
# invent topics can also invent a typo and publish into it for weeks with nobody
# noticing. And auto-created topics take the broker defaults, which is how a
# topic that should be compacted and retained for a year ends up deleted after
# seven days.
#
# One topic per aggregate. Partition count is the ceiling on consumer
# parallelism and cannot be lowered later, so it is set for the traffic
# expected, not for the traffic today.
apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
name: classic-models.catalog
namespace: kafka
labels:
strimzi.io/cluster: kafka
spec:
partitions: 3
replicas: 1
config:
retention.ms: "604800000" # 7 days
cleanup.policy: delete
min.insync.replicas: "1"
---
apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
name: classic-models.sales
namespace: kafka
labels:
strimzi.io/cluster: kafka
spec:
partitions: 3
replicas: 1
config:
retention.ms: "604800000"
cleanup.policy: delete
min.insync.replicas: "1"
---
apiVersion: kafka.strimzi.io/v1
kind: KafkaTopic
metadata:
name: classic-models.payments
namespace: kafka
labels:
strimzi.io/cluster: kafka
spec:
partitions: 3
replicas: 1
config:
retention.ms: "604800000"
cleanup.policy: delete
min.insync.replicas: "1"
kafka-users.yaml
+34 -59
View File
@@ -52,64 +52,39 @@ spec:
operations:
- Describe
- Write
---
# The classic-models API, which publishes domain events and consumes nothing.
#
# Write-only and topic-scoped: a producer that can also read every topic is a
# producer that can exfiltrate every other domain's events if it is ever
# compromised. Describe is needed for metadata; it does not grant reads.
apiVersion: kafka.strimzi.io/v1
kind: KafkaUser
metadata:
name: classic-models-api
namespace: kafka
labels:
strimzi.io/cluster: kafka
spec:
authentication:
type: scram-sha-512
authorization:
type: simple
acls:
- resource:
type: topic
name: classic-models.
patternType: prefix
operations:
- Write
- Describe
---
# The Approvisionnement agent, which is woken by catalog events.
# --- adding an application account ----------------------------------------
#
# Read on one topic and one consumer-group prefix, nothing else. It cannot see
# sales or payments — an agent whose job is stock has no business reading who
# paid what — and it cannot write, so a compromised workflow can raise no event
# that another agent would then act on.
apiVersion: kafka.strimzi.io/v1
kind: KafkaUser
metadata:
name: classic-models-agent-approvisionnement
namespace: kafka
labels:
strimzi.io/cluster: kafka
spec:
authentication:
type: scram-sha-512
authorization:
type: simple
acls:
- resource:
type: topic
name: classic-models.catalog
patternType: literal
operations:
- Read
- Describe
- resource:
type: group
name: approvisionnement-
patternType: prefix
operations:
- Read
- Describe
# Do not add one here. An account belongs to the application that uses it: its
# own repo says what it may read and write, and adding or removing the
# application adds or removes its access. This repo deploys a cluster and holds
# no opinion about who uses it.
#
# Create the CR from the application's own repo, in **this** namespace — Strimzi
# runs with `watchAnyNamespace=false`, so a KafkaUser anywhere else is ignored,
# silently. Copy the shape below and cut it down to what the application does:
#
# apiVersion: kafka.strimzi.io/v1
# kind: KafkaUser
# metadata:
# name: <application>
# namespace: kafka
# labels:
# strimzi.io/cluster: kafka
# spec:
# authentication:
# type: scram-sha-512
# authorization:
# type: simple
# acls:
# - resource: {type: topic, name: <prefix>, patternType: prefix}
# operations: [Write, Describe] # a producer
# - resource: {type: group, name: <prefix>, patternType: prefix}
# operations: [Read, Describe] # a consumer's group
#
# A producer that can also read every topic exfiltrates every other
# application's data if it is ever compromised, and a consumer that can write
# can raise an event someone else acts on. Grant one direction, on one prefix.
#
# `kafka-client` above is the wide-open template: useful for a smoke test, and
# for nothing that runs unattended.