stt/tech-watch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e1577fe06113c83595ce86009b304c82d980fc27
tech-watch/README.md
T
sttlab e1577fe061 first commit
2026-07-22 14:41:10 +00:00

287 lines
14 KiB
Markdown
Raw Blame History

# Tech Watch Automation Stack — Miniflux + n8n + digest workflow
This covers the full stack described in `AGENTS.md` (namespace `watch`,
shared PostgreSQL in namespace `postgres`, Miniflux with the initial feed
bootstrap, n8n for workflow orchestration, and the daily LLM digest
workflow) plus a small read-only file server for the generated digests.
Everything below is deployed and active on this cluster.
## Layout
```
k8s/
namespace.yaml # namespace: watch (apps)
postgres/ # shared PostgreSQL instance (namespace: postgres)
namespace.yaml
secret.example.yaml # superuser credentials
statefulset.yaml
service.yaml
miniflux/ # namespace: watch (except create-db-job.yaml)
secret.example.yaml # DATABASE_URL, admin credentials
deployment.yaml
service.yaml
gateway.yaml # Gateway API (Envoy), HTTPS listener on 443
httproute.yaml # routes miniflux.sttlab.pc to the Service
create-db-job.yaml # runs in ns postgres: creates the "miniflux" database
bootstrap-feeds-job.yaml # runs miniflux-initializer to create categories/feeds
n8n/ # namespace: watch (except create-db-job.yaml)
secret.example.yaml # DB credentials, N8N_ENCRYPTION_KEY
miniflux-token-secret.example.yaml # Miniflux API token, backup/reference for the n8n credential
pvc.yaml # n8n-data (n8n's own /home/node/.n8n) + n8n-digest-workspace (/data, for Read/Write File nodes)
deployment.yaml
service.yaml
gateway.yaml # Gateway API (Envoy), HTTPS listener on 443
httproute.yaml # routes n8n.sttlab.pc to the Service
create-db-job.yaml # runs in ns postgres: creates the "n8n" database
digest-files/ # namespace: watch — read-only nginx file server
configmap.yaml # nginx.conf: autoindex on
deployment.yaml # mounts n8n-digest-workspace PVC (digests/ subdir), read-only
service.yaml
gateway.yaml # routes digests.sttlab.pc
httproute.yaml
miniflux-initializer/ # container image used by bootstrap-feeds-job.yaml
Dockerfile
bootstrap_feeds.py
feeds.yaml
requirements.txt
prompts/
digest.md # LLM system prompt for the digest workflow; mounted into n8n via
# the n8n-digest-prompt ConfigMap (regenerate after editing, see below)
workflows/
digest.json # "Tech Watch Digest" n8n workflow (daily 06:00 Europe/Paris cron)
```
## Deploy
1. **Namespaces**
```
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/postgres/namespace.yaml
```
2. **PostgreSQL** — `k8s/postgres/{namespace,secret.example,statefulset,service}.yaml`
deploy a standalone instance and are kept for reference/portability, but
**on this cluster a shared PostgreSQL already runs** in the `postgres`
namespace (Bitnami Helm chart, release `postgres`, service
`postgres-postgresql`). Reuse it instead of applying the StatefulSet:
```
cp k8s/postgres/secret.example.yaml k8s/postgres/secret.yaml
# edit k8s/postgres/secret.yaml: POSTGRES_PASSWORD must match the real
# instance's password:
# kubectl get secret postgres-postgresql -n postgres -o jsonpath='{.data.postgres-password}' | base64 -d
kubectl apply -f k8s/postgres/secret.yaml
kubectl apply -f k8s/miniflux/create-db-job.yaml
kubectl wait --for=condition=complete job/create-db-miniflux -n postgres --timeout=60s
```
(If deploying to a cluster with no existing shared Postgres, apply
`statefulset.yaml`/`service.yaml` first and update
`create-db-job.yaml`'s host from `postgres-postgresql` to `postgres`.)
3. **TLS secret** for the Gateway listeners (using the existing wildcard cert for `*.sttlab.pc`)
```
kubectl create secret tls sttlab-pc-tls \
--cert=~/tls/sttlab.pc.crt --key=~/tls/sttlab.pc.key \
-n watch
```
4. **Miniflux**
```
cp k8s/miniflux/secret.example.yaml k8s/miniflux/secret.yaml
# edit k8s/miniflux/secret.yaml:
# - DATABASE_URL password must match k8s/postgres/secret.yaml
# - set ADMIN_USERNAME / ADMIN_PASSWORD for the Miniflux admin account
kubectl apply -f k8s/miniflux/secret.yaml
kubectl apply -f k8s/miniflux/deployment.yaml
kubectl apply -f k8s/miniflux/service.yaml
kubectl apply -f k8s/miniflux/gateway.yaml
kubectl apply -f k8s/miniflux/httproute.yaml
```
Exposed via Gateway API (Envoy Gateway, `gatewayClassName: envoy`) on
port 443 — this cluster's `ingress-nginx` only exposes ports 80/9443
externally, not 443, so a plain `Ingress` would not actually be
reachable over HTTPS. Point DNS for `miniflux.sttlab.pc` at the Gateway
address if not already done (`kubectl get gateway miniflux -n watch`).
Feed `crawler` (fetch original article content instead of the raw
RSS/Atom excerpt) is enabled on all feeds except OpenAI's — its blog is
behind a Cloudflare bot challenge that always rejects Miniflux's
fetcher, so it stays on the RSS excerpt. See `crawler:` per feed in
`miniflux-initializer/feeds.yaml`.
5. **Bootstrap feeds** — runs as a one-off Job using a locally built image
(no external registry needed on a single-node k3s cluster):
```
cd miniflux-initializer
docker build --network=host -t miniflux-initializer:latest .
docker save miniflux-initializer:latest | sudo k3s ctr images import -
cd ..
kubectl apply -f k8s/miniflux/bootstrap-feeds-job.yaml
kubectl wait --for=condition=complete job/bootstrap-feeds -n watch --timeout=60s
kubectl logs -n watch job/bootstrap-feeds
```
Safe to re-run: delete the Job (`kubectl delete job bootstrap-feeds -n
watch`) and re-apply after editing `miniflux-initializer/feeds.yaml`
(rebuild + re-import the image first if you changed it). Only adds new
feeds — it does not update `crawler`/settings on feeds that already
exist (use the Miniflux API or UI for that).
> Note: `--network=host` is required for the `pip install` step during
> the image build — the local router does not answer DNS queries from
> Docker's bridge subnet, only the host's own resolver works.
6. **API token for n8n**: create it manually in the Miniflux UI under
*Settings → API Keys* — there is no public REST endpoint to create one.
Kept as the source of truth in `k8s/n8n/miniflux-token-secret.yaml` (see
`.example` for the format); the n8n credential itself is created in step
8 below, from the same token.
7. **n8n**
```
cp k8s/n8n/secret.example.yaml k8s/n8n/secret.yaml
# edit k8s/n8n/secret.yaml:
# - DB_POSTGRESDB_PASSWORD must match k8s/postgres/secret.yaml
# - N8N_ENCRYPTION_KEY: generate once with `openssl rand -hex 24` and
# never rotate it afterwards (it decrypts every saved credential)
kubectl apply -f k8s/n8n/secret.yaml
kubectl apply -f k8s/n8n/create-db-job.yaml
kubectl wait --for=condition=complete job/create-db-n8n -n postgres --timeout=60s
kubectl apply -f k8s/n8n/pvc.yaml
# ConfigMap holding the digest LLM prompt (prompts/digest.md is the source
# of truth; regenerate and re-apply this any time the file changes):
kubectl create configmap n8n-digest-prompt --from-file=digest.md=prompts/digest.md \
-n watch --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/n8n/deployment.yaml
kubectl apply -f k8s/n8n/service.yaml
kubectl apply -f k8s/n8n/gateway.yaml
kubectl apply -f k8s/n8n/httproute.yaml
```
Same Gateway API setup as Miniflux (TLS terminated by Envoy Gateway on
443, reusing the `sttlab-pc-tls` secret). Point DNS for `n8n.sttlab.pc`
at the Gateway address once (`kubectl get gateway n8n -n watch`).
On first login, n8n prompts to create the owner account (email/password)
— there's no `CREATE_ADMIN`-style env var like Miniflux, so this step is
manual in the UI.
Two PVCs are mounted: `n8n-data` is n8n's own `/home/node/.n8n` (config,
binary data cache); `n8n-digest-workspace` is mounted separately at
`/data` for the digest workflow's Read/Write File nodes (prompt +
generated digests). They're kept apart because n8n unconditionally
blocks file-node access to its own `/home/node/.n8n` directory
(`N8N_BLOCK_FILE_ACCESS_TO_N8N_FILES`, not overridable via
`N8N_RESTRICT_FILE_ACCESS_TO`) — a dedicated volume avoids that
restriction honestly instead of working around it. Note also that
`N8N_RESTRICT_FILE_ACCESS_TO` only accepts a single path, not a list —
hence both `prompts/` and `digests/` live under the same `/data` root.
8. **Digest workflow** — import `workflows/digest.json` (the "Tech Watch
Digest" workflow: daily 06:00 Europe/Paris cron) and wire its
credentials via the n8n Public API (Settings → n8n API → create a key
first):
```
N8N_KEY=<your n8n API key>
# Miniflux credential (Header Auth, X-Auth-Token) — use the same token
# as k8s/n8n/miniflux-token-secret.yaml
curl -s https://n8n.sttlab.pc/api/v1/credentials -X POST \
-H "X-N8N-API-KEY: $N8N_KEY" -H "Content-Type: application/json" \
-d '{"name":"Miniflux API","type":"httpHeaderAuth","data":{"name":"X-Auth-Token","value":"<token>","allowedHttpRequestDomains":"all"}}'
# LLM credential (Header Auth, Authorization: Bearer <key>) — OpenRouter or
# any OpenAI-compatible provider
curl -s https://n8n.sttlab.pc/api/v1/credentials -X POST \
-H "X-N8N-API-KEY: $N8N_KEY" -H "Content-Type: application/json" \
-d '{"name":"LLM API","type":"httpHeaderAuth","data":{"name":"Authorization","value":"Bearer <key>","allowedHttpRequestDomains":"all"}}'
# Import the workflow, then edit the two credential IDs in its HTTP
# Request nodes ("Get Unread Entries" / "Mark Entries As Read" -> Miniflux
# API; "Summarize Category" -> LLM API) to match the IDs returned above,
# and set the real llmBaseUrl / llmModel in the "Init" node's code.
curl -s https://n8n.sttlab.pc/api/v1/workflows -X POST \
-H "X-N8N-API-KEY: $N8N_KEY" -H "Content-Type: application/json" \
--data-binary @<(python3 -c "import json; wf=json.load(open('workflows/digest.json')); print(json.dumps({k: wf[k] for k in ('name','nodes','connections','settings')}))")
# Activate once credentials are wired and a manual run succeeds:
curl -s https://n8n.sttlab.pc/api/v1/workflows/<id>/activate -X POST \
-H "X-N8N-API-KEY: $N8N_KEY"
```
The workflow uses Miniflux's **internal** cluster Service DNS
(`http://miniflux.watch.svc.cluster.local/v1/entries`), not the external
HTTPS hostname — `miniflux.sttlab.pc` isn't resolvable from inside the
cluster (it only exists as a Gateway route, no DNS record).
To test-run before activating (the running Deployment pod can't run
`n8n execute` itself — its Task Broker port is already bound — so use a
throwaway Job with the same image/env/volumes instead), or just use the
"Test workflow" button in the n8n UI from the Schedule Trigger node.
Pipeline: fetch unread Miniflux entries from the last 24h → group by
category → for each category, call the LLM (system prompt from
`prompts/digest.md`) to pick the 3-5 most relevant items + weak signals,
strict JSON output → compose one Markdown file → write it to
`/data/digests/digest-<date>.md` → mark the processed entries as read in
Miniflux.
9. **Digest file server** — serves `/data/digests/` (read-only) so digests
can be browsed/downloaded from a browser instead of email:
```
kubectl apply -f k8s/digest-files/
```
Same Gateway/TLS pattern, routes `digests.sttlab.pc`. Point DNS at the
Gateway address (`kubectl get gateway digest-files -n watch`).
## How to add a source
Add an entry under the relevant category (or a new category) in
`miniflux-initializer/feeds.yaml`, verify the URL responds with `curl -I
<url>`, then rebuild the image and re-run the bootstrap Job (see step 5).
## Validate manifests
```
kubectl apply --dry-run=client -f k8s/namespace.yaml
kubectl apply --dry-run=client -f k8s/postgres/
kubectl apply --dry-run=client -f k8s/miniflux/
kubectl apply --dry-run=client -f k8s/n8n/
kubectl apply --dry-run=client -f k8s/digest-files/
```
(Run against a cluster with the `watch`/`postgres` namespaces and secrets
already created, since some manifests reference them.)
## Smoke-test checklist
- [ ] `kubectl get pods -n postgres` — postgres pod `Running`, `1/1` ready
- [ ] `kubectl get pods -n watch` — miniflux pod `Running`, `1/1` ready
- [ ] `kubectl logs -n watch deploy/miniflux` shows migrations applied, no errors
- [ ] `https://miniflux.sttlab.pc/healthcheck` returns `OK`
- [ ] Log in to the Miniflux UI with the admin credentials from the secret
- [ ] `kubectl logs -n watch job/bootstrap-feeds` shows categories/feeds created
- [ ] Categories and feeds appear in the Miniflux UI
- [ ] At least one feed shows fetched entries (Miniflux polls periodically; use "Refresh" in the UI to force it)
- [ ] `GET /v1/entries?status=unread` (with an API token) returns entries
- [ ] `kubectl get pods -n watch` — n8n pod `Running`, `1/1` ready
- [ ] `kubectl logs -n watch deploy/n8n` shows no DB connection errors
- [ ] `https://n8n.sttlab.pc/healthz` returns `{"status":"ok"}`
- [ ] Log in to the n8n UI and create the owner account
- [ ] n8n can reach Miniflux: an HTTP Request node to
`http://miniflux.watch.svc.cluster.local/v1/me` with the API token
returns the admin user
- [ ] "Tech Watch Digest" workflow imported, both credentials (Miniflux API,
LLM API) attached to their HTTP Request nodes
- [ ] A manual run ("Test workflow" in the UI, or a throwaway
`n8n execute --id=<id>` Job) produces a file under `/data/digests/`
in the n8n pod, and the corresponding Miniflux entries are marked read
- [ ] Workflow activated (`"active": true`)
- [ ] `kubectl get pods -n watch` — digest-files pod `Running`, `1/1` ready
- [ ] `https://digests.sttlab.pc/` lists the generated digest files
## Current status
Deployed and active: Miniflux (9 feeds), n8n, "Tech Watch Digest" workflow
(daily 06:00 Europe/Paris, OpenRouter `deepseek/deepseek-v4-flash`), and
the digest file server at `https://digests.sttlab.pc/`. No email delivery
(SMTP not configured) — digests are Markdown files served over HTTP
instead.
Reference in New Issue View Git Blame Copy Permalink