transcriptor-ai/CLAUDE.md

transcriptor-ai

Local, no-cloud meeting transcription for macOS. See README.md for what this is, how to install dependencies, and how to run it — this file is for working on the code: decisions, reasoning, conventions, and current implementation status.

Design/decision history lives in a long chat transcript, not in this repo yet. This file is the distilled result. If something here seems arbitrary, assume there was a reason discussed elsewhere rather than re-litigating it from scratch — but flag it if it looks wrong given the current code.

Goal

Primary: follow a meeting's transcript live while it happens, so the user can stay focused on listening instead of note-taking. Meetings mix French and English mid-sentence (code-switching), include non-native English accents, and technical/internal jargon that should be correctable via a small per-meeting glossary.

Out of scope for now, do not pull forward without being asked:

• Real-time content analysis / suggested questions during the meeting (explicitly phase 2).
• Post-processing (fuzzy-match glossary correction, LLM reread via llama.cpp). The pipeline currently publishes qwen-asr's raw output straight to the SSE broadcaster. Revisit only when asked — don't propose implementing this preemptively, it's been deferred twice.

Only live transcription (capture → VAD → ASR → SSE) is in scope today.

Target hardware

MacBook (base) M3, 24 GB unified memory. Not a high-end Max/Pro chip — size and quantize models accordingly (prefer 4-bit; avoid anything like an 8B+ reread model). Do not assume generous headroom. A separate M4 Max/128GB machine exists for development but is not the deployment target; don't let it justify heavier defaults.

Architecture

audiotee (subprocess, 2 outputs)
  ├─ stdout        → system audio track (PCM)
  └─ --mic-output  → mic track (PCM, via a FIFO for live streaming, not a plain file)
        │
        ▼ (per track, independently)
  VAD (simple energy-threshold heuristic, hand-rolled, no ML dependency for v1)
        │  chunks with overlap
        ▼
  ASR: qwen-asr subprocess (Qwen3-ASR-1.7B, --stream, --prompt for glossary biasing)
        │  timestamped segments: {track, text, is_final, timestamp}
        ▼
  merge (chronological, across both tracks)
        │
        ▼
  SSE endpoint (served by this Go binary) — terminal client is the first consumer

Post-processing (fuzzy-match glossary correction, then LLM reread via llama.cpp subprocess) was in the original target architecture between merge and SSE, but is explicitly deferred — see "Out of scope for now" above. Raw qwen-asr output is published as-is today.

Why these choices (don't re-decide without reason)

• Two tracks, two independent VAD+ASR pipelines (one per track), not one merged audio stream. This is a cheap "me vs them" diarization proxy — no real diarization model needed. Rejected merging into a single flow: it was considered and reverted, keep 2+2.
• Go, not Python or Swift, for this orchestrator. All the actual ML work (ASR, LLM reread) is delegated to external subprocess binaries regardless of orchestrator language, so the orchestrator's job is just: spawn subprocesses, pipe bytes, run a small HTTP/SSE server, merge by timestamp. That's Go's home turf (os/exec, net/http stdlib, single static binary, no runtime dependency). Python was rejected for its dependency/environment management overhead; Swift was rejected because its ML ecosystem (Silero VAD, MLX model loading for this specific model) is immature enough to require significant custom porting work that buys nothing here. Rust was a close second to Go but adds ownership/borrowing ceremony with no corresponding benefit for I/O-bound glue code.
• VAD: hand-rolled energy-threshold heuristic, not Silero, for v1. No Swift/Go ML ecosystem is mature enough here to justify the integration cost. Upgrade only if segmentation quality turns out to be the actual bottleneck — verify before adding complexity.
• ASR runtime: qwen-asr (https://github.com/antirez/qwen-asr), a pure-C implementation with --stream and --prompt (glossary biasing) already built in. Consumed as a subprocess, one instance per track. Build is trivial on macOS: no dependency install needed beyond the OS's built-in Accelerate framework (make blas). No prebuilt binaries are published, so it's vendored as a git submodule (third_party/qwen-asr, pinned to a specific commit) and built by scripts/build-dist.sh — this exists specifically so transcriptor-ui (the planned native app, see "Related repos") can produce a fully self-contained, one-click-install .app with no separate manual clone/build step for the end user; models are the only thing still fetched at runtime, deliberately (multi-GB, doesn't belong baked into a build). Verified empirically: --stream does not print incremental partial text to stdout for a single bounded stdin stream — only the final transcription, once done. Our own VAD segment boundaries are what stand in for "live" granularity, not qwen-asr's internal streaming. One subprocess call per VAD segment (measured ~0.15-0.4x realtime including process startup and model load, on both the 0.6B and 1.7B models) is fast enough — no need for a persistent daemon process.
• LLM reread runtime: llama.cpp, installed via brew install llama.cpp (prebuilt, official Homebrew formula — no source build needed). Model: small, 4-bit quantized, sized for the 24GB target machine, not the dev machine. Not wired in yet (deferred).
• Live display: SSE (Server-Sent Events), not WebSocket. The data flow is one-directional (server → clients); SSE is plain HTTP, trivially consumable by a terminal client and later by a browser (EventSource, no client library needed). WebSocket would be over-engineering for a need that has no bidirectional requirement today.
• No Docker. Ruled out on technical grounds, not preference: Docker Desktop on macOS runs a Linux VM with no Metal GPU passthrough, which breaks any Metal-accelerated inference, and Core Audio access from inside a container is impractical.

Message format (SSE)

One JSON object per transcript segment (defined in internal/transcript):

{"track": "system", "text": "...", "is_final": false, "timestamp": "2026-08-07T14:32:01.123Z"}

is_final: false = a streaming ASR partial hypothesis that may still be revised. is_final: true = the segment is closed (VAD ended the utterance).

No timestamp-based reordering across tracks yet — messages are broadcast in the order transcribeTrack produces them. Measured ASR latency is small relative to segment spacing (3-5s), so this hasn't been observed to cause visible out-of-order delivery. Add a small reordering buffer in internal/transcript if it ever becomes a real problem — not before.

Related repos

../audiotee — the Swift CLI this project spawns as a subprocess for audio capture (see internal/capture). Consumed as a subprocess deliberately, not imported as a Swift library (AudioTeeCore is exposed as one, but that only matters for a Swift consumer, which this isn't) — see its own CONTEXT.md §6.2 for why that repo's TCC-identity signing work matters specifically because of this. Two tracks (system audio + mic) go to two separate outputs (stdout + a FIFO), never multiplexed onto one stream — two concurrent Core Audio IO threads writing raw PCM chunks to a shared fd risked interleaving/corruption.

../transcriptor-ui — a native macOS SwiftUI app (separate Xcode project), the "one-click" way to actually use this — built and working as of 2026-08-08, not just planned anymore. It spawns transcriptor-ai as a subprocess (same pattern as transcriptor-ai → audiotee — each layer manages the one below it via a subprocess + a clean protocol, never merged into one project/toolchain) and is an SSE client of it, same as transcript-tail, just with a native UI instead of a terminal. Its Xcode build phase builds this repo's scripts/build-dist.sh and ../audiotee's scripts/build-signed.sh and embeds all three binaries in the .app bundle — verified fully self-contained (tested with ~/bin/audiotee moved aside, capture still worked from the embedded copy). Deliberately not a monorepo with this one — see "Why these choices": mixing Go and Swift tooling in one Xcode project would undermine the reason Go was chosen for this repo in the first place. transcriptor-ui will eventually let the user pick ASR model/options and configure the glossary and post-processing (question-detection, meeting-summary generation via a local LLM) — model selection is still a dev-time hardcoded path there, nothing else is wired up yet; check transcriptor-ui's own CLAUDE.md for its current state rather than assuming.

-audiotee-binary flag exists specifically for transcriptor-ui's benefit: without it, internal/capture only finds audiotee via PATH then ~/bin/audiotee, which is fine for CLI use but wrong for a bundled app that must not depend on anything pre-installed outside its own .app — transcriptor-ui passes its embedded copy's path explicitly.

Conventions

• Go idioms: gofmt/go vet clean, standard project layout (cmd/, internal/).
• Prefer the standard library over third-party Go modules where it's a reasonably close fit (os/exec, net/http, encoding/json) — the "self-contained binary, no dependency bazar" property is a deliberate, stated goal of this project, not an accident.
• Code and comments in English. Conversation with the user in French (matches the sibling audiotee repo's convention).
• Don't reintroduce Python, Swift, Docker, or WebSocket for this project without a new, explicit reason — see "Why these choices" above.
• User-facing install/usage docs belong in README.md, not here — keep this file about reasoning and internal state, not instructions for running the tool.

Status

Capture → VAD → ASR → SSE is implemented and verified end to end:

• internal/capture — audiotee subprocess wrapper (stdout + mic FIFO).
• internal/vad — energy-threshold segmentation.
• internal/asr — one qwen_asr subprocess call per segment.
• internal/transcript — Broadcaster (SSE), FileWriter (durable text file, one line per message, fsync'd on every write), Message format shared by both.
• cmd/transcriptor-ai — wires all of the above together; currently also the test harness.
• cmd/transcript-tail — minimal SSE terminal client. Exits calmly (code 0, "stream ended (server stopped?)") when the server shuts down, instead of an alarming raw connection error.
• scripts/build-dist.sh — builds qwen_asr (from the third_party/qwen-asr submodule) and transcriptor-ai into dist/, ready for transcriptor-ui to embed. Verified: builds cleanly from the submodule, and the resulting dist/qwen_asr actually transcribes (smoke-tested), not just "it compiled."
• -capture-system/-capture-mic flags (transcriptor-ui's benefit: a UI toggle for what to capture). Asymmetric by necessity, document this if it ever confuses someone: -capture-mic=false skips mic capture at the audiotee level entirely (no --capture-mic passed, no mic permission requested) — internal/capture.Config.CaptureMic controls this. -capture-system=false cannot do the equivalent — audiotee has no flag to skip its system tap, it's always running — so system-only-disabled just means transcriptor-ai receives system chunks (VAD still runs on them, negligible cost) but never dispatches them to an ASR worker (main.go's dispatch helper drops them). Verified all three modes (system-only, mic-only, both) via CLI: mic-only correctly showed zero track: system messages even with system audio playing, confirming the drop works, not just the flag parsing.

Decision (2026-08-08): the future native macOS UI (transcriptor-ui, a separate repo/Xcode project — see below) will not write the transcript file itself; transcriptor-ai does, via -transcript-file, specifically because it's the more crash-resistant of the two (simpler control flow than a UI process) and the whole point of this file is surviving a crash.

Verified: real human speech (English + French code-switching) through an actual microphone, with both qwen3-asr-0.6b (fast iteration) and qwen3-asr-1.7b (the target model). 1.7B measured at ~0.4x realtime (2.3s to process a 5.9s clip, model load included) — comfortably fast enough for one-subprocess-per-segment. Switching models is a runtime flag (-asr-model-dir), not a rebuild.

Three concurrency/signal-handling bugs were found and fixed via real testing (not just code review):

• transcribeWorker (then still per-track, named transcribeTrack) was using the SIGINT-canceled context for still-queued transcriptions (fixed with a separate background context for ASR calls).

• main() wasn't waiting for the ASR worker goroutine(s) to finish before exiting (fixed with a sync.WaitGroup).

• internal/capture.Stop() called cmd.Wait() before the pipe-reading goroutines had finished draining, which os/exec docs call out as incorrect and can truncate the last bit of audio.

• qwen_asr subprocesses were killed by the terminal's own Ctrl+C, independent of any of the above: child processes inherit the parent's process group by default, and a real terminal's Ctrl+C sends SIGINT to the whole foreground process group — not just to transcriptor-ai. So an in-flight qwen_asr call died with "signal: interrupt" regardless of the context-handling fix above, which only guards against our own code cancelling it. Fixed in internal/asr/runner.go with SysProcAttr.Setpgid: true, putting each qwen_asr in its own process group. This bug was invisible to kill -INT <transcriptor-ai-pid>, the way earlier manual tests in this session were done — that only signals one PID, not a process group, so it never reproduced the real terminal behavior. Verified instead by sending SIGINT to the process group (os.killpg from a small Python harness, or kill -INT -<pgid> — careful with the latter run from an interactive shell, it'll also hit the shell itself if unrelated jobs share its group).

• Duplicate text at force-cut segment boundaries, found from real usage on a real meeting-like recording (documentary audio) — visibly bad enough that the transcript was unusable, e.g. "à vivre" transcribed once at the end of a segment and again at the start of the next. Root cause: the 300ms Overlap audio carried into the next force-cut segment (for ASR context continuity) was being fully re-transcribed and re-published, so both copies showed up in the live transcript. Fixed by defaulting vad.Config.Overlap to 0 — accept an occasional word cut in half at a hard boundary rather than systematic duplication, which is far worse for actually reading the transcript. Found and fixed alongside it: tail() in internal/vad/vad.go treated n <= 0 as "return the whole buffer" instead of empty, which would have made Overlap: 0 carry over the entire previous segment instead of nothing — fixed to return nil for n <= 0. If overlap is ever reintroduced, it needs text-level dedup on the published output, not just raw audio carry-over.

• System and mic tracks transcribing the same content — not a code bug, a structural limit of tapping raw Core Audio input: if audio plays through speakers (not headphones), the mic picks up the acoustic leakage and transcribes it too. Conferencing apps do their own acoustic echo cancellation on the signal they send to other participants, but that doesn't touch what our raw mic tap sees. Real-world implication: headphones are likely a hard requirement for the two-track "me vs them" diarization to actually hold during real meetings, not just a nice-to-have — unresolved, needs a decision (require headphones, or build real AEC — a much bigger feature) rather than a quick fix.

• Shutdown could feel unresponsive during a large in-flight transcription backlog (no feedback while waiting, especially on the slower 1.7B model). Fixed with a "shutting down, waiting for in-flight transcriptions..." message and a second Ctrl+C now force-exits immediately instead of making the user wait through the full drain.

ASR memory: single shared worker instead of one per track (2026-08-10). The 1.7B model ("large" in transcriptor-ui) has no quantized-weights option — qwen_asr only supports BF16 (see third_party/qwen-asr/README.md, "Memory Requirements": ~6.9GiB static footprint for the 1.7B model). Before this fix, cmd/transcriptor-ai/main.go ran one transcribeTrack goroutine per track (system, mic), each independently spawning qwen_asr subprocesses — so whenever both tracks had speech at the same time, two 1.7B instances ran concurrently, measured peaking at ~13GiB combined, over half of the 24GB target machine's unified memory (see "Target hardware"). Fixed by merging both tracks onto one shared worker/channel (asrSegs, transcribeWorker — renamed from transcribeTrack since it's no longer per-track): dispatch() now sends every enabled track's segments onto the same channel instead of two separate ones, so qwen_asr invocations are strictly serialized regardless of which track they came from. Per-track chronological order is preserved (VAD's own output channel is already chronological across both tracks; the worker just drains it in that order). Trade-off: if both tracks talk at once, one track's transcription now waits for the other's to finish instead of running in parallel — accepted, since the memory pressure was the more pressing constraint on the 24GB target machine. Verified empirically, not just "it compiles": ran the real pipeline with both tracks capturing simultaneous audio (a video through system output, say for spoken input) and polled pgrep -x qwen_asr at 0.3s intervals throughout — max concurrent qwen_asr processes observed was 1, confirmed never 2, while both tracks still produced correctly-ordered transcript segments.

Not yet implemented: post-processing (deferred, see "Out of scope for now"), and anything from audiotee's CONTEXT.md §6.4-adjacent concerns beyond what's listed here.

Resume point (2026-08-07): the live pipeline is done and validated on real speech with both models, and the force-cut duplication bug is fixed and verified. The system/mic acoustic leakage issue (headphones vs AEC) is still an open decision. No specific next task was chosen before pausing beyond that — ask the user rather than assuming which.