first commit

2026-07-22 14:41:10 +00:00
commit e1577fe061
35 changed files with 1847 additions and 0 deletions
@@ -0,0 +1,218 @@
+# AGENTS.md — Tech Watch Automation Stack
+
+Instructions for implementing an automated technology-watch (veille) pipeline on Kubernetes.
+
+## Goal
+
+Deploy a self-hosted stack that:
+1. Aggregates tech news sources (RSS and APIs) via **Miniflux**
+2. Orchestrates workflows via **n8n** (fetch unread articles, deduplicate, summarize, deliver digest)
+3. Uses an **LLM through an OpenAI-compatible API** (OpenRouter today, possibly a local LLM proxy later — never hardcode OpenRouter specifics)
+4. Produces a **weekly (and optionally daily) digest**, prioritized by topic, delivered by email and/or saved as Markdown
+
+## Architecture
+
+```
+[RSS/API sources] --> [Miniflux] --(REST API)--> [n8n workflows] --> [LLM (OpenAI-compatible)] --> [Digest: email / file / webhook]
+                          |                          |
+                     [PostgreSQL] <------------------+  (n8n also uses Postgres)
+```
+
+## Stack & Constraints
+
+- **Kubernetes** deployment (target: any conformant cluster). Prefer **Helm charts** where official/community charts exist; otherwise write plain manifests (Kustomize layout welcome).
+- **Miniflux**: official container image `miniflux/miniflux`. Needs PostgreSQL. Enable API access. Run DB migrations via env `RUN_MIGRATIONS=1`.
+- **PostgreSQL**: one instance (or one per app if simpler), e.g. Bitnami chart or CloudNativePG. Persistent volumes required.
+- **n8n**: official image `n8nio/n8n`. Configure with Postgres backend (not SQLite). Persist `/home/node/.n8n`.
+- **LLM access**: strictly through the **OpenAI-compatible chat completions interface** (`/v1/chat/completions`). Configuration via env vars only:
+  - `LLM_BASE_URL` (e.g. `https://openrouter.ai/api/v1`, later a local proxy URL)
+  - `LLM_API_KEY`
+  - `LLM_MODEL` (model identifier string, treat as opaque)
+  - Do not use vendor-specific SDK features; plain HTTP calls or the generic OpenAI node in n8n with a custom base URL.
+- **Secrets**: Kubernetes Secrets (no secrets in git). Provide `*.example` files.
+- **Ingress**: expose Miniflux and n8n UIs behind an Ingress (TLS assumed handled by cluster / cert-manager). Make hostnames configurable.
+- **Code and comments in English.**
+
+## Implementation Tasks
+
+1. **Manifests/Charts**
+   - Namespace `veille` (configurable)
+   - PostgreSQL with PVC, credentials in Secret
+   - Miniflux Deployment + Service + Ingress; admin credentials in Secret; `DATABASE_URL` from Secret
+   - n8n Deployment + Service + Ingress; `DB_TYPE=postgresdb` config; encryption key in Secret
+   - NetworkPolicies (optional, nice to have): only n8n → Miniflux, apps → Postgres
+2. **Miniflux bootstrap**
+   - Script or Job that creates the feed list (see Sources below) via the Miniflux REST API (`POST /v1/feeds`), organized in categories matching the topics
+   - Create an API token for n8n
+3. **n8n workflows** (export as JSON in repo, `workflows/`)
+   - **Digest workflow** (cron, weekly Friday 07:00 Europe/Paris; optional daily variant):
+     1. `GET /v1/entries?status=unread&limit=...` from Miniflux
+     2. Group entries by category/topic
+     3. For each topic: call LLM to summarize + rank top items (relevance to the topics listed below); one LLM call per topic to control context size
+     4. Compose final digest (Markdown): per topic — 3–5 top items with 1–2 sentence summaries + links, then a short "weak signals" section
+     5. Deliver: send email (SMTP node) and/or commit/save Markdown; then mark entries as read (`PUT /v1/entries` status=read)
+   - **Error handling**: retries on HTTP calls, dead-letter notification on failure
+4. **Docs**
+   - `README.md`: deployment steps, secret setup, how to add a source, how to change LLM endpoint
+5. **Validation**
+   - Lint manifests (`kubectl apply --dry-run=client` or `kubeconform`)
+   - Provide a smoke-test checklist
+
+## Digest Priorities (for LLM prompt)
+
+Rank content by relevance to, in order:
+1. Application & data integration (iPaaS, ESB, API management)
+2. Event-driven architecture, messaging, streaming (Kafka, AMQP, MQTT, AsyncAPI, CloudEvents)
+3. Agentic AI, LLM integration patterns, MCP (Model Context Protocol)
+4. Cloud native / Kubernetes / platform engineering
+5. MFT (Managed File Transfer) and B2B/EDI integration
+6. General cloud computing announcements (only if significant)
+
+## Sources to Configure in Miniflux
+
+All sources below are RSS/Atom unless stated otherwise. If a feed URL is broken at implementation time, locate the current one (feeds move); as a fallback use an HTML-to-RSS bridge (e.g. RSSHub) or n8n HTTP scraping.
+
+### Cloud Native / Kubernetes
+| Source | Access | URL |
+|---|---|---|
+| CNCF blog | RSS | https://www.cncf.io/feed/ |
+| Kubernetes blog | RSS | https://kubernetes.io/feed.xml |
+| The New Stack | RSS | https://thenewstack.io/feed/ |
+| InfoQ – Cloud | RSS | https://feed.infoq.com/cloud-computing/ |
+| InfoQ – DevOps | RSS | https://feed.infoq.com/devops/ |
+
+### Cloud Providers
+| Source | Access | URL |
+|---|---|---|
+| AWS News Blog | RSS | https://aws.amazon.com/blogs/aws/feed/ |
+| AWS Compute/Integration blogs | RSS | discover per-category feeds under aws.amazon.com/blogs/ |
+| Azure updates | RSS | https://azurecomm.azure.com/updates/feed/ (verify; fallback: Azure blog https://azure.microsoft.com/en-us/blog/feed/) |
+| Google Cloud blog | RSS | https://cloudblog.withgoogle.com/rss/ |
+
+### Integration / iPaaS / API
+| Source | Access | URL |
+|---|---|---|
+| MuleSoft blog | RSS | https://blogs.mulesoft.com/feed/ |
+| Boomi blog | RSS | https://boomi.com/blog/feed/ (verify) |
+| Kong blog | RSS | https://konghq.com/blog/rss.xml (verify) |
+| Apache Camel blog | RSS | https://camel.apache.org/feed.xml (verify) |
+| WSO2 blog | RSS | https://wso2.com/feed/ (verify) |
+
+### EDA / Messaging / Streaming
+| Source | Access | URL |
+|---|---|---|
+| Confluent blog | RSS | https://www.confluent.io/blog/feed/ (verify) |
+| Kai Waehner | RSS | https://www.kai-waehner.de/blog/feed/ |
+| Solace blog | RSS | https://solace.com/feed/ |
+| AsyncAPI blog | RSS | https://www.asyncapi.com/rss.xml (verify) |
+| RabbitMQ blog | RSS | https://www.rabbitmq.com/blog/rss.xml (verify) |
+| Apache Kafka releases | Atom (GitHub) | https://github.com/apache/kafka/releases.atom |
+
+### Agentic AI / LLM
+| Source | Access | URL |
+|---|---|---|
+| Anthropic news | RSS | https://www.anthropic.com/rss.xml (verify; fallback RSSHub) |
+| OpenAI blog | RSS | https://openai.com/blog/rss.xml (verify) |
+| LangChain blog | RSS | https://blog.langchain.dev/rss/ |
+| Simon Willison | RSS | https://simonwillison.net/atom/everything/ |
+| Latent Space | RSS | https://www.latent.space/feed |
+| MCP spec repo releases | Atom (GitHub) | https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom (verify repo path) |
+
+### MFT / B2B / EDI
+| Source | Access | URL |
+|---|---|---|
+| Axway blog | RSS | https://blog.axway.com/feed (verify) |
+| Cleo blog | RSS | https://www.cleo.com/blog/rss.xml (verify) |
+| SEEBURGER blog | RSS | https://blog.seeburger.com/feed/ (verify) |
+| IBM Sterling / integration topics | RSS or scrape | discover on ibm.com/blog; fallback RSSHub |
+
+### Cross-cutting / Signal
+| Source | Access | URL |
+|---|---|---|
+| Hacker News (front page) | RSS | https://news.ycombinator.com/rss |
+| HN best (less noise) | RSS | https://hnrss.org/best |
+| ThoughtWorks Technology Radar | RSS | https://www.thoughtworks.com/rss/insights.xml (verify) — biannual, high value |
+| Reddit r/devops | RSS | https://www.reddit.com/r/devops/.rss |
+| Reddit r/kubernetes | RSS | https://www.reddit.com/r/kubernetes/.rss |
+| Reddit r/dataengineering | RSS | https://www.reddit.com/r/dataengineering/.rss |
+
+### Security / Vulnerabilities
+| Source | Access | URL |
+|---|---|---|
+| CISA Known Exploited Vulnerabilities | RSS/JSON | https://www.cisa.gov/cybersecurity-advisories/all.xml (advisories); KEV catalog JSON: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json (poll via n8n) |
+| Kubernetes security announces | Google Group RSS | https://groups.google.com/g/kubernetes-security-announce (verify RSS availability; fallback: kubernetes.io blog security tag) |
+| GitHub Security Advisories | Atom per ecosystem/repo | https://github.com/advisories (filterable; per-repo: `https://github.com/{org}/{repo}/security/advisories` — poll via n8n/API) |
+| The Hacker News | RSS | https://feeds.feedburner.com/TheHackersNews |
+| Bleeping Computer | RSS | https://www.bleepingcomputer.com/feed/ |
+| NVD CVE feed | JSON API | https://services.nvd.nist.gov/rest/json/cves/2.0 — poll via n8n with keyword filters (kafka, kubernetes, n8n, miniflux, postgresql…); too noisy for raw RSS |
+
+Digest handling: security items form a dedicated "Security" section in the digest, filtered by relevance to the deployed/covered stack (Kubernetes, Kafka, PostgreSQL, n8n, integration middleware). Critical/KEV items may trigger an immediate notification instead of waiting for the weekly digest.
+
+### Non-RSS sources (handled in n8n, not Miniflux)
+- **GitHub releases** for key projects: use `https://github.com/{org}/{repo}/releases.atom` (works in Miniflux) — add repos as needed (kubernetes, istio, keda, camel, kafka, langchain, n8n…)
+- **Google Alerts**: create alerts ("MFT managed file transfer", "EDI modernization", "iPaaS"), deliver as RSS, add to Miniflux
+- **LinkedIn / Gartner summaries**: no reliable API; keep manual
+
+### Market Trends / Business (M&A, funding, analyst views)
+| Source | Access | URL |
+|---|---|---|
+| TechCrunch – Enterprise | RSS | https://techcrunch.com/category/enterprise/feed/ |
+| The Register | RSS | https://www.theregister.com/headlines.atom |
+| SiliconANGLE | RSS | https://siliconangle.com/feed/ (enterprise/cloud coverage, theCUBE analysts) |
+| Futurum Group | RSS | https://futurumgroup.com/feed/ (verify) — accessible analyst research |
+| Constellation Research blog | RSS | https://www.constellationr.com/rss.xml (verify) |
+| CNCF Annual Survey & reports | Scrape/manual | https://www.cncf.io/reports/ — annual, high value |
+| DB-Engines ranking | Scrape/manual | https://db-engines.com/en/ranking — monthly trend check via n8n |
+| Gartner / Forrester | Manual | paywalled; capture via vendors' "named leader" announcements (already covered by vendor blog feeds) and analyst LinkedIn posts |
+
+Annual reports worth flagging in the digest when released: CNCF Survey, Stack Overflow Developer Survey, DORA State of DevOps, ThoughtWorks Tech Radar (biannual), State of the API (Postman).
+
+Digest handling: market items go in a dedicated "Market & Trends" section — acquisitions, funding rounds, major partnerships, analyst rankings — restricted to the integration/EDA/cloud/AI space. Example of relevant signal: the IBM acquisition of Confluent ($11B, 2026).
+
+### People to Follow
+For each person: prefer their **personal blog RSS** if one exists (check first); otherwise create a **Google Alert** on their quoted name, delivered as RSS, and add it to Miniflux in a "People" category.
+
+**Integration / iPaaS / API**
+- Kai Waehner — blog RSS exists (already listed above); independent since leaving Confluent (2026)
+- Sanjeev Mohan (data & analytics, ex-Gartner)
+- Massimo Pezzini (ex-Gartner, iPaaS/HIP reference)
+- Saurabh Sharma (integration analyst)
+- Holger Mueller (Constellation Research)
+
+**EDA / Messaging / Streaming**
+- Gwen Shapira (Kafka)
+- Adam Bellemare (event-driven microservices)
+- Hugo Guerrero (API/messaging, Red Hat)
+- Clemens Vasters (Microsoft, messaging/CloudEvents)
+
+**Agentic AI / LLM**
+- Simon Willison — blog RSS exists (already listed above)
+- Andrew Ng
+- Harrison Chase (LangChain)
+- swyx / Shawn Wang (Latent Space — RSS already listed)
+- Chip Huyen (ML systems)
+
+**Cloud native / Platform engineering**
+- Kelsey Hightower
+- Viktor Farcic (DevOps Toolkit)
+- Sam Newman (microservices)
+- Charity Majors (observability)
+
+**Architecture**
+- Gregor Hohpe (Enterprise Integration Patterns) — check for blog RSS (architectelevator.com)
+- Neal Ford (ThoughtWorks)
+- Mark Richards (developertoasarchitect.com)
+
+**MFT / B2B / EDI**: no strong public figures; the iPaaS analysts above (Pezzini, Sharma) also cover B2B.
+
+
+## Notes for the Agent
+
+- **Truncated feeds**: many feeds (media, vendor blogs) only carry title + excerpt, not full content. Strategy:
+  1. Enable Miniflux **"Fetch original content"** (scraper) per feed for truncated sources — set `crawler: true` when creating feeds via the API (test each feed; disable if the site blocks scraping).
+  2. In the n8n digest workflow, treat `content` defensively: if the entry content is below ~500 characters, either summarize from title + excerpt only, or fetch the article URL directly (HTTP node) as fallback. Never assume full text is available.
+- Verify every feed URL with an HTTP GET before adding it; log and replace dead feeds.
+- Keep the LLM prompt for summarization in a dedicated file (`prompts/digest.md`) so it can be tuned without touching workflows.
+- Digest language: **French** output for the digest content; everything else (code, comments, docs) in English.
+- Keep resource requests modest (this is a personal stack): e.g. 256Mi/0.1 CPU requests per app, adjust limits sensibly.
+- Timezone: Europe/Paris for all cron schedules.