tech-watch/AGENTS.md

# AGENTS.md — Tech Watch Automation Stack

Instructions for implementing an automated technology-watch (veille) pipeline on Kubernetes.

## Goal

Deploy a self-hosted stack that:
1. Aggregates tech news sources (RSS and APIs) via **Miniflux**
2. Orchestrates workflows via **n8n** (fetch unread articles, deduplicate, summarize, deliver digest)
3. Uses an **LLM through an OpenAI-compatible API** (OpenRouter today, possibly a local LLM proxy later — never hardcode OpenRouter specifics)
4. Produces a **weekly (and optionally daily) digest**, prioritized by topic, delivered by email and/or saved as Markdown

## Architecture

```
[RSS/API sources] --> [Miniflux] --(REST API)--> [n8n workflows] --> [LLM (OpenAI-compatible)] --> [Digest: email / file / webhook]
                          |                          |
                     [PostgreSQL] <------------------+  (n8n also uses Postgres)
```

## Stack & Constraints

- **Kubernetes** deployment (target: any conformant cluster). Prefer **Helm charts** where official/community charts exist; otherwise write plain manifests (Kustomize layout welcome).
- **Miniflux**: official container image `miniflux/miniflux`. Needs PostgreSQL. Enable API access. Run DB migrations via env `RUN_MIGRATIONS=1`.
- **PostgreSQL**: one instance (or one per app if simpler), e.g. Bitnami chart or CloudNativePG. Persistent volumes required.
- **n8n**: official image `n8nio/n8n`. Configure with Postgres backend (not SQLite). Persist `/home/node/.n8n`.
- **LLM access**: strictly through the **OpenAI-compatible chat completions interface** (`/v1/chat/completions`). Configuration via env vars only:
  - `LLM_BASE_URL` (e.g. `https://openrouter.ai/api/v1`, later a local proxy URL)
  - `LLM_API_KEY`
  - `LLM_MODEL` (model identifier string, treat as opaque)
  - Do not use vendor-specific SDK features; plain HTTP calls or the generic OpenAI node in n8n with a custom base URL.
- **Secrets**: Kubernetes Secrets (no secrets in git). Provide `*.example` files.
- **Ingress**: expose Miniflux and n8n UIs behind an Ingress (TLS assumed handled by cluster / cert-manager). Make hostnames configurable.
- **Code and comments in English.**

## Implementation Tasks

1. **Manifests/Charts**
   - Namespace `veille` (configurable)
   - PostgreSQL with PVC, credentials in Secret
   - Miniflux Deployment + Service + Ingress; admin credentials in Secret; `DATABASE_URL` from Secret
   - n8n Deployment + Service + Ingress; `DB_TYPE=postgresdb` config; encryption key in Secret
   - NetworkPolicies (optional, nice to have): only n8n → Miniflux, apps → Postgres
2. **Miniflux bootstrap**
   - Script or Job that creates the feed list (see Sources below) via the Miniflux REST API (`POST /v1/feeds`), organized in categories matching the topics
   - Create an API token for n8n
3. **n8n workflows** (export as JSON in repo, `workflows/`)
   - **Digest workflow** (cron, weekly Friday 07:00 Europe/Paris; optional daily variant):
     1. `GET /v1/entries?status=unread&limit=...` from Miniflux
     2. Group entries by category/topic
     3. For each topic: call LLM to summarize + rank top items (relevance to the topics listed below); one LLM call per topic to control context size
     4. Compose final digest (Markdown): per topic — 3–5 top items with 1–2 sentence summaries + links, then a short "weak signals" section
     5. Deliver: send email (SMTP node) and/or commit/save Markdown; then mark entries as read (`PUT /v1/entries` status=read)
   - **Error handling**: retries on HTTP calls, dead-letter notification on failure
4. **Docs**
   - `README.md`: deployment steps, secret setup, how to add a source, how to change LLM endpoint
5. **Validation**
   - Lint manifests (`kubectl apply --dry-run=client` or `kubeconform`)
   - Provide a smoke-test checklist

## Digest Priorities (for LLM prompt)

Rank content by relevance to, in order:
1. Application & data integration (iPaaS, ESB, API management)
2. Event-driven architecture, messaging, streaming (Kafka, AMQP, MQTT, AsyncAPI, CloudEvents)
3. Agentic AI, LLM integration patterns, MCP (Model Context Protocol)
4. Cloud native / Kubernetes / platform engineering
5. MFT (Managed File Transfer) and B2B/EDI integration
6. General cloud computing announcements (only if significant)

## Sources to Configure in Miniflux

All sources below are RSS/Atom unless stated otherwise. If a feed URL is broken at implementation time, locate the current one (feeds move); as a fallback use an HTML-to-RSS bridge (e.g. RSSHub) or n8n HTTP scraping.

### Cloud Native / Kubernetes
| Source | Access | URL |
|---|---|---|
| CNCF blog | RSS | https://www.cncf.io/feed/ |
| Kubernetes blog | RSS | https://kubernetes.io/feed.xml |
| The New Stack | RSS | https://thenewstack.io/feed/ |
| InfoQ – Cloud | RSS | https://feed.infoq.com/cloud-computing/ |
| InfoQ – DevOps | RSS | https://feed.infoq.com/devops/ |

### Cloud Providers
| Source | Access | URL |
|---|---|---|
| AWS News Blog | RSS | https://aws.amazon.com/blogs/aws/feed/ |
| AWS Compute/Integration blogs | RSS | discover per-category feeds under aws.amazon.com/blogs/ |
| Azure updates | RSS | https://azurecomm.azure.com/updates/feed/ (verify; fallback: Azure blog https://azure.microsoft.com/en-us/blog/feed/) |
| Google Cloud blog | RSS | https://cloudblog.withgoogle.com/rss/ |

### Integration / iPaaS / API
| Source | Access | URL |
|---|---|---|
| MuleSoft blog | RSS | https://blogs.mulesoft.com/feed/ |
| Boomi blog | RSS | https://boomi.com/blog/feed/ (verify) |
| Kong blog | RSS | https://konghq.com/blog/rss.xml (verify) |
| Apache Camel blog | RSS | https://camel.apache.org/feed.xml (verify) |
| WSO2 blog | RSS | https://wso2.com/feed/ (verify) |

### EDA / Messaging / Streaming
| Source | Access | URL |
|---|---|---|
| Confluent blog | RSS | https://www.confluent.io/blog/feed/ (verify) |
| Kai Waehner | RSS | https://www.kai-waehner.de/blog/feed/ |
| Solace blog | RSS | https://solace.com/feed/ |
| AsyncAPI blog | RSS | https://www.asyncapi.com/rss.xml (verify) |
| RabbitMQ blog | RSS | https://www.rabbitmq.com/blog/rss.xml (verify) |
| Apache Kafka releases | Atom (GitHub) | https://github.com/apache/kafka/releases.atom |

### Agentic AI / LLM
| Source | Access | URL |
|---|---|---|
| Anthropic news | RSS | https://www.anthropic.com/rss.xml (verify; fallback RSSHub) |
| OpenAI blog | RSS | https://openai.com/blog/rss.xml (verify) |
| LangChain blog | RSS | https://blog.langchain.dev/rss/ |
| Simon Willison | RSS | https://simonwillison.net/atom/everything/ |
| Latent Space | RSS | https://www.latent.space/feed |
| MCP spec repo releases | Atom (GitHub) | https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom (verify repo path) |

### MFT / B2B / EDI
| Source | Access | URL |
|---|---|---|
| Axway blog | RSS | https://blog.axway.com/feed (verify) |
| Cleo blog | RSS | https://www.cleo.com/blog/rss.xml (verify) |
| SEEBURGER blog | RSS | https://blog.seeburger.com/feed/ (verify) |
| IBM Sterling / integration topics | RSS or scrape | discover on ibm.com/blog; fallback RSSHub |

### Cross-cutting / Signal
| Source | Access | URL |
|---|---|---|
| Hacker News (front page) | RSS | https://news.ycombinator.com/rss |
| HN best (less noise) | RSS | https://hnrss.org/best |
| ThoughtWorks Technology Radar | RSS | https://www.thoughtworks.com/rss/insights.xml (verify) — biannual, high value |
| Reddit r/devops | RSS | https://www.reddit.com/r/devops/.rss |
| Reddit r/kubernetes | RSS | https://www.reddit.com/r/kubernetes/.rss |
| Reddit r/dataengineering | RSS | https://www.reddit.com/r/dataengineering/.rss |

### Security / Vulnerabilities
| Source | Access | URL |
|---|---|---|
| CISA Known Exploited Vulnerabilities | RSS/JSON | https://www.cisa.gov/cybersecurity-advisories/all.xml (advisories); KEV catalog JSON: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json (poll via n8n) |
| Kubernetes security announces | Google Group RSS | https://groups.google.com/g/kubernetes-security-announce (verify RSS availability; fallback: kubernetes.io blog security tag) |
| GitHub Security Advisories | Atom per ecosystem/repo | https://github.com/advisories (filterable; per-repo: `https://github.com/{org}/{repo}/security/advisories` — poll via n8n/API) |
| The Hacker News | RSS | https://feeds.feedburner.com/TheHackersNews |
| Bleeping Computer | RSS | https://www.bleepingcomputer.com/feed/ |
| NVD CVE feed | JSON API | https://services.nvd.nist.gov/rest/json/cves/2.0 — poll via n8n with keyword filters (kafka, kubernetes, n8n, miniflux, postgresql…); too noisy for raw RSS |

Digest handling: security items form a dedicated "Security" section in the digest, filtered by relevance to the deployed/covered stack (Kubernetes, Kafka, PostgreSQL, n8n, integration middleware). Critical/KEV items may trigger an immediate notification instead of waiting for the weekly digest.

### Non-RSS sources (handled in n8n, not Miniflux)
- **GitHub releases** for key projects: use `https://github.com/{org}/{repo}/releases.atom` (works in Miniflux) — add repos as needed (kubernetes, istio, keda, camel, kafka, langchain, n8n…)
- **Google Alerts**: create alerts ("MFT managed file transfer", "EDI modernization", "iPaaS"), deliver as RSS, add to Miniflux
- **LinkedIn / Gartner summaries**: no reliable API; keep manual

### Market Trends / Business (M&A, funding, analyst views)
| Source | Access | URL |
|---|---|---|
| TechCrunch – Enterprise | RSS | https://techcrunch.com/category/enterprise/feed/ |
| The Register | RSS | https://www.theregister.com/headlines.atom |
| SiliconANGLE | RSS | https://siliconangle.com/feed/ (enterprise/cloud coverage, theCUBE analysts) |
| Futurum Group | RSS | https://futurumgroup.com/feed/ (verify) — accessible analyst research |
| Constellation Research blog | RSS | https://www.constellationr.com/rss.xml (verify) |
| CNCF Annual Survey & reports | Scrape/manual | https://www.cncf.io/reports/ — annual, high value |
| DB-Engines ranking | Scrape/manual | https://db-engines.com/en/ranking — monthly trend check via n8n |
| Gartner / Forrester | Manual | paywalled; capture via vendors' "named leader" announcements (already covered by vendor blog feeds) and analyst LinkedIn posts |

Annual reports worth flagging in the digest when released: CNCF Survey, Stack Overflow Developer Survey, DORA State of DevOps, ThoughtWorks Tech Radar (biannual), State of the API (Postman).

Digest handling: market items go in a dedicated "Market & Trends" section — acquisitions, funding rounds, major partnerships, analyst rankings — restricted to the integration/EDA/cloud/AI space. Example of relevant signal: the IBM acquisition of Confluent ($11B, 2026).

### People to Follow
For each person: prefer their **personal blog RSS** if one exists (check first); otherwise create a **Google Alert** on their quoted name, delivered as RSS, and add it to Miniflux in a "People" category.

**Integration / iPaaS / API**
- Kai Waehner — blog RSS exists (already listed above); independent since leaving Confluent (2026)
- Sanjeev Mohan (data & analytics, ex-Gartner)
- Massimo Pezzini (ex-Gartner, iPaaS/HIP reference)
- Saurabh Sharma (integration analyst)
- Holger Mueller (Constellation Research)

**EDA / Messaging / Streaming**
- Gwen Shapira (Kafka)
- Adam Bellemare (event-driven microservices)
- Hugo Guerrero (API/messaging, Red Hat)
- Clemens Vasters (Microsoft, messaging/CloudEvents)

**Agentic AI / LLM**
- Simon Willison — blog RSS exists (already listed above)
- Andrew Ng
- Harrison Chase (LangChain)
- swyx / Shawn Wang (Latent Space — RSS already listed)
- Chip Huyen (ML systems)

**Cloud native / Platform engineering**
- Kelsey Hightower
- Viktor Farcic (DevOps Toolkit)
- Sam Newman (microservices)
- Charity Majors (observability)

**Architecture**
- Gregor Hohpe (Enterprise Integration Patterns) — check for blog RSS (architectelevator.com)
- Neal Ford (ThoughtWorks)
- Mark Richards (developertoasarchitect.com)

**MFT / B2B / EDI**: no strong public figures; the iPaaS analysts above (Pezzini, Sharma) also cover B2B.


## Notes for the Agent

- **Truncated feeds**: many feeds (media, vendor blogs) only carry title + excerpt, not full content. Strategy:
  1. Enable Miniflux **"Fetch original content"** (scraper) per feed for truncated sources — set `crawler: true` when creating feeds via the API (test each feed; disable if the site blocks scraping).
  2. In the n8n digest workflow, treat `content` defensively: if the entry content is below ~500 characters, either summarize from title + excerpt only, or fetch the article URL directly (HTTP node) as fallback. Never assume full text is available.
- Verify every feed URL with an HTTP GET before adding it; log and replace dead feeds.
- Keep the LLM prompt for summarization in a dedicated file (`prompts/digest.md`) so it can be tuned without touching workflows.
- Digest language: **French** output for the digest content; everything else (code, comments, docs) in English.
- Keep resource requests modest (this is a personal stack): e.g. 256Mi/0.1 CPU requests per app, adjust limits sensibly.
- Timezone: Europe/Paris for all cron schedules.