stt/tech-watch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
tech-watch/AGENTS.md
T
sttlab e1577fe061 first commit
2026-07-22 14:41:10 +00:00

13 KiB
Raw Permalink Blame History

AGENTS.md — Tech Watch Automation Stack

Instructions for implementing an automated technology-watch (veille) pipeline on Kubernetes.

Goal

Deploy a self-hosted stack that:

  1. Aggregates tech news sources (RSS and APIs) via Miniflux
  2. Orchestrates workflows via n8n (fetch unread articles, deduplicate, summarize, deliver digest)
  3. Uses an LLM through an OpenAI-compatible API (OpenRouter today, possibly a local LLM proxy later — never hardcode OpenRouter specifics)
  4. Produces a weekly (and optionally daily) digest, prioritized by topic, delivered by email and/or saved as Markdown

Architecture

[RSS/API sources] --> [Miniflux] --(REST API)--> [n8n workflows] --> [LLM (OpenAI-compatible)] --> [Digest: email / file / webhook]
                          |                          |
                     [PostgreSQL] <------------------+  (n8n also uses Postgres)

Stack & Constraints

  • Kubernetes deployment (target: any conformant cluster). Prefer Helm charts where official/community charts exist; otherwise write plain manifests (Kustomize layout welcome).
  • Miniflux: official container image miniflux/miniflux. Needs PostgreSQL. Enable API access. Run DB migrations via env RUN_MIGRATIONS=1.
  • PostgreSQL: one instance (or one per app if simpler), e.g. Bitnami chart or CloudNativePG. Persistent volumes required.
  • n8n: official image n8nio/n8n. Configure with Postgres backend (not SQLite). Persist /home/node/.n8n.
  • LLM access: strictly through the OpenAI-compatible chat completions interface (/v1/chat/completions). Configuration via env vars only:
    • LLM_BASE_URL (e.g. https://openrouter.ai/api/v1, later a local proxy URL)
    • LLM_API_KEY
    • LLM_MODEL (model identifier string, treat as opaque)
    • Do not use vendor-specific SDK features; plain HTTP calls or the generic OpenAI node in n8n with a custom base URL.
  • Secrets: Kubernetes Secrets (no secrets in git). Provide *.example files.
  • Ingress: expose Miniflux and n8n UIs behind an Ingress (TLS assumed handled by cluster / cert-manager). Make hostnames configurable.
  • Code and comments in English.

Implementation Tasks

  1. Manifests/Charts
    • Namespace veille (configurable)
    • PostgreSQL with PVC, credentials in Secret
    • Miniflux Deployment + Service + Ingress; admin credentials in Secret; DATABASE_URL from Secret
    • n8n Deployment + Service + Ingress; DB_TYPE=postgresdb config; encryption key in Secret
    • NetworkPolicies (optional, nice to have): only n8n → Miniflux, apps → Postgres
  2. Miniflux bootstrap
    • Script or Job that creates the feed list (see Sources below) via the Miniflux REST API (POST /v1/feeds), organized in categories matching the topics
    • Create an API token for n8n
  3. n8n workflows (export as JSON in repo, workflows/)
    • Digest workflow (cron, weekly Friday 07:00 Europe/Paris; optional daily variant):
      1. GET /v1/entries?status=unread&limit=... from Miniflux
      2. Group entries by category/topic
      3. For each topic: call LLM to summarize + rank top items (relevance to the topics listed below); one LLM call per topic to control context size
      4. Compose final digest (Markdown): per topic — 3–5 top items with 1–2 sentence summaries + links, then a short "weak signals" section
      5. Deliver: send email (SMTP node) and/or commit/save Markdown; then mark entries as read (PUT /v1/entries status=read)
    • Error handling: retries on HTTP calls, dead-letter notification on failure
  4. Docs
    • README.md: deployment steps, secret setup, how to add a source, how to change LLM endpoint
  5. Validation
    • Lint manifests (kubectl apply --dry-run=client or kubeconform)
    • Provide a smoke-test checklist

Digest Priorities (for LLM prompt)

Rank content by relevance to, in order:

  1. Application & data integration (iPaaS, ESB, API management)
  2. Event-driven architecture, messaging, streaming (Kafka, AMQP, MQTT, AsyncAPI, CloudEvents)
  3. Agentic AI, LLM integration patterns, MCP (Model Context Protocol)
  4. Cloud native / Kubernetes / platform engineering
  5. MFT (Managed File Transfer) and B2B/EDI integration
  6. General cloud computing announcements (only if significant)

Sources to Configure in Miniflux

All sources below are RSS/Atom unless stated otherwise. If a feed URL is broken at implementation time, locate the current one (feeds move); as a fallback use an HTML-to-RSS bridge (e.g. RSSHub) or n8n HTTP scraping.

Cloud Native / Kubernetes

Source Access URL
CNCF blog RSS https://www.cncf.io/feed/
Kubernetes blog RSS https://kubernetes.io/feed.xml
The New Stack RSS https://thenewstack.io/feed/
InfoQ – Cloud RSS https://feed.infoq.com/cloud-computing/
InfoQ – DevOps RSS https://feed.infoq.com/devops/

Cloud Providers

Source Access URL
AWS News Blog RSS https://aws.amazon.com/blogs/aws/feed/
AWS Compute/Integration blogs RSS discover per-category feeds under aws.amazon.com/blogs/
Azure updates RSS https://azurecomm.azure.com/updates/feed/ (verify; fallback: Azure blog https://azure.microsoft.com/en-us/blog/feed/)
Google Cloud blog RSS https://cloudblog.withgoogle.com/rss/

Integration / iPaaS / API

Source Access URL
MuleSoft blog RSS https://blogs.mulesoft.com/feed/
Boomi blog RSS https://boomi.com/blog/feed/ (verify)
Kong blog RSS https://konghq.com/blog/rss.xml (verify)
Apache Camel blog RSS https://camel.apache.org/feed.xml (verify)
WSO2 blog RSS https://wso2.com/feed/ (verify)

EDA / Messaging / Streaming

Source Access URL
Confluent blog RSS https://www.confluent.io/blog/feed/ (verify)
Kai Waehner RSS https://www.kai-waehner.de/blog/feed/
Solace blog RSS https://solace.com/feed/
AsyncAPI blog RSS https://www.asyncapi.com/rss.xml (verify)
RabbitMQ blog RSS https://www.rabbitmq.com/blog/rss.xml (verify)
Apache Kafka releases Atom (GitHub) https://github.com/apache/kafka/releases.atom

Agentic AI / LLM

Source Access URL
Anthropic news RSS https://www.anthropic.com/rss.xml (verify; fallback RSSHub)
OpenAI blog RSS https://openai.com/blog/rss.xml (verify)
LangChain blog RSS https://blog.langchain.dev/rss/
Simon Willison RSS https://simonwillison.net/atom/everything/
Latent Space RSS https://www.latent.space/feed
MCP spec repo releases Atom (GitHub) https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom (verify repo path)

MFT / B2B / EDI

Source Access URL
Axway blog RSS https://blog.axway.com/feed (verify)
Cleo blog RSS https://www.cleo.com/blog/rss.xml (verify)
SEEBURGER blog RSS https://blog.seeburger.com/feed/ (verify)
IBM Sterling / integration topics RSS or scrape discover on ibm.com/blog; fallback RSSHub

Cross-cutting / Signal

Source Access URL
Hacker News (front page) RSS https://news.ycombinator.com/rss
HN best (less noise) RSS https://hnrss.org/best
ThoughtWorks Technology Radar RSS https://www.thoughtworks.com/rss/insights.xml (verify) — biannual, high value
Reddit r/devops RSS https://www.reddit.com/r/devops/.rss
Reddit r/kubernetes RSS https://www.reddit.com/r/kubernetes/.rss
Reddit r/dataengineering RSS https://www.reddit.com/r/dataengineering/.rss

Security / Vulnerabilities

Source Access URL
CISA Known Exploited Vulnerabilities RSS/JSON https://www.cisa.gov/cybersecurity-advisories/all.xml (advisories); KEV catalog JSON: https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json (poll via n8n)
Kubernetes security announces Google Group RSS https://groups.google.com/g/kubernetes-security-announce (verify RSS availability; fallback: kubernetes.io blog security tag)
GitHub Security Advisories Atom per ecosystem/repo https://github.com/advisories (filterable; per-repo: https://github.com/{org}/{repo}/security/advisories — poll via n8n/API)
The Hacker News RSS https://feeds.feedburner.com/TheHackersNews
Bleeping Computer RSS https://www.bleepingcomputer.com/feed/
NVD CVE feed JSON API https://services.nvd.nist.gov/rest/json/cves/2.0 — poll via n8n with keyword filters (kafka, kubernetes, n8n, miniflux, postgresql…); too noisy for raw RSS

Digest handling: security items form a dedicated "Security" section in the digest, filtered by relevance to the deployed/covered stack (Kubernetes, Kafka, PostgreSQL, n8n, integration middleware). Critical/KEV items may trigger an immediate notification instead of waiting for the weekly digest.

Non-RSS sources (handled in n8n, not Miniflux)

  • GitHub releases for key projects: use https://github.com/{org}/{repo}/releases.atom (works in Miniflux) — add repos as needed (kubernetes, istio, keda, camel, kafka, langchain, n8n…)
  • Google Alerts: create alerts ("MFT managed file transfer", "EDI modernization", "iPaaS"), deliver as RSS, add to Miniflux
  • LinkedIn / Gartner summaries: no reliable API; keep manual

Market Trends / Business (M&A, funding, analyst views)

Source Access URL
TechCrunch – Enterprise RSS https://techcrunch.com/category/enterprise/feed/
The Register RSS https://www.theregister.com/headlines.atom
SiliconANGLE RSS https://siliconangle.com/feed/ (enterprise/cloud coverage, theCUBE analysts)
Futurum Group RSS https://futurumgroup.com/feed/ (verify) — accessible analyst research
Constellation Research blog RSS https://www.constellationr.com/rss.xml (verify)
CNCF Annual Survey & reports Scrape/manual https://www.cncf.io/reports/ — annual, high value
DB-Engines ranking Scrape/manual https://db-engines.com/en/ranking — monthly trend check via n8n
Gartner / Forrester Manual paywalled; capture via vendors' "named leader" announcements (already covered by vendor blog feeds) and analyst LinkedIn posts

Annual reports worth flagging in the digest when released: CNCF Survey, Stack Overflow Developer Survey, DORA State of DevOps, ThoughtWorks Tech Radar (biannual), State of the API (Postman).

Digest handling: market items go in a dedicated "Market & Trends" section — acquisitions, funding rounds, major partnerships, analyst rankings — restricted to the integration/EDA/cloud/AI space. Example of relevant signal: the IBM acquisition of Confluent ($11B, 2026).

People to Follow

For each person: prefer their personal blog RSS if one exists (check first); otherwise create a Google Alert on their quoted name, delivered as RSS, and add it to Miniflux in a "People" category.

Integration / iPaaS / API

  • Kai Waehner — blog RSS exists (already listed above); independent since leaving Confluent (2026)
  • Sanjeev Mohan (data & analytics, ex-Gartner)
  • Massimo Pezzini (ex-Gartner, iPaaS/HIP reference)
  • Saurabh Sharma (integration analyst)
  • Holger Mueller (Constellation Research)

EDA / Messaging / Streaming

  • Gwen Shapira (Kafka)
  • Adam Bellemare (event-driven microservices)
  • Hugo Guerrero (API/messaging, Red Hat)
  • Clemens Vasters (Microsoft, messaging/CloudEvents)

Agentic AI / LLM

  • Simon Willison — blog RSS exists (already listed above)
  • Andrew Ng
  • Harrison Chase (LangChain)
  • swyx / Shawn Wang (Latent Space — RSS already listed)
  • Chip Huyen (ML systems)

Cloud native / Platform engineering

  • Kelsey Hightower
  • Viktor Farcic (DevOps Toolkit)
  • Sam Newman (microservices)
  • Charity Majors (observability)

Architecture

  • Gregor Hohpe (Enterprise Integration Patterns) — check for blog RSS (architectelevator.com)
  • Neal Ford (ThoughtWorks)
  • Mark Richards (developertoasarchitect.com)

MFT / B2B / EDI: no strong public figures; the iPaaS analysts above (Pezzini, Sharma) also cover B2B.

Notes for the Agent

  • Truncated feeds: many feeds (media, vendor blogs) only carry title + excerpt, not full content. Strategy:
    1. Enable Miniflux "Fetch original content" (scraper) per feed for truncated sources — set crawler: true when creating feeds via the API (test each feed; disable if the site blocks scraping).
    2. In the n8n digest workflow, treat content defensively: if the entry content is below ~500 characters, either summarize from title + excerpt only, or fetch the article URL directly (HTTP node) as fallback. Never assume full text is available.
  • Verify every feed URL with an HTTP GET before adding it; log and replace dead feeds.
  • Keep the LLM prompt for summarization in a dedicated file (prompts/digest.md) so it can be tuned without touching workflows.
  • Digest language: French output for the digest content; everything else (code, comments, docs) in English.
  • Keep resource requests modest (this is a personal stack): e.g. 256Mi/0.1 CPU requests per app, adjust limits sensibly.
  • Timezone: Europe/Paris for all cron schedules.
Reference in New Issue View Git Blame Copy Permalink