transcriptor-ai/CLAUDE.md

# transcriptor-ai

Local, no-cloud meeting transcription for macOS. See `README.md` for what this is, how to
install dependencies, and how to run it — this file is for working on the code: decisions,
reasoning, conventions, and current implementation status.

Design/decision history lives in a long chat transcript, not in this repo yet. This file is
the distilled result. If something here seems arbitrary, assume there was a reason discussed
elsewhere rather than re-litigating it from scratch — but flag it if it looks wrong given the
current code.

## Goal

Primary: follow a meeting's transcript live while it happens, so the user can stay focused on
listening instead of note-taking. Meetings mix French and English mid-sentence
(code-switching), include non-native English accents, and technical/internal jargon that
should be correctable via a small per-meeting glossary.

**Out of scope for now, do not pull forward without being asked:**
- Real-time content analysis / suggested questions during the meeting (explicitly phase 2).
- Post-processing (fuzzy-match glossary correction, LLM reread via llama.cpp). The pipeline
  currently publishes qwen-asr's raw output straight to the SSE broadcaster. Revisit only
  when asked — don't propose implementing this preemptively, it's been deferred twice.

Only live transcription (capture → VAD → ASR → SSE) is in scope today.

## Target hardware

**MacBook (base) M3, 24 GB unified memory.** Not a high-end Max/Pro chip — size and quantize
models accordingly (prefer 4-bit; avoid anything like an 8B+ reread model). Do not assume
generous headroom. A separate M4 Max/128GB machine exists for development but is not the
deployment target; don't let it justify heavier defaults.

## Architecture

```
audiotee (subprocess, 2 outputs)
  ├─ stdout        → system audio track (PCM)
  └─ --mic-output  → mic track (PCM, via a FIFO for live streaming, not a plain file)
        │
        ▼ (per track, independently)
  VAD (simple energy-threshold heuristic, hand-rolled, no ML dependency for v1)
        │  chunks with overlap
        ▼
  ASR: qwen-asr subprocess (Qwen3-ASR-1.7B, --stream, --prompt for glossary biasing)
        │  timestamped segments: {track, text, is_final, timestamp}
        ▼
  merge (chronological, across both tracks)
        │
        ▼
  SSE endpoint (served by this Go binary) — terminal client is the first consumer
```

Post-processing (fuzzy-match glossary correction, then LLM reread via llama.cpp subprocess)
was in the original target architecture between merge and SSE, but is explicitly deferred —
see "Out of scope for now" above. Raw qwen-asr output is published as-is today.

### Why these choices (don't re-decide without reason)

- **Two tracks, two independent VAD+ASR pipelines** (one per track), not one merged audio
  stream. This is a cheap "me vs them" diarization proxy — no real diarization model needed.
  Rejected merging into a single flow: it was considered and reverted, keep 2+2.
- **Go**, not Python or Swift, for this orchestrator. All the actual ML work (ASR, LLM
  reread) is delegated to external subprocess binaries regardless of orchestrator language,
  so the orchestrator's job is just: spawn subprocesses, pipe bytes, run a small HTTP/SSE
  server, merge by timestamp. That's Go's home turf (`os/exec`, `net/http` stdlib, single
  static binary, no runtime dependency). Python was rejected for its dependency/environment
  management overhead; Swift was rejected because its ML ecosystem (Silero VAD, MLX model
  loading for this specific model) is immature enough to require significant custom porting
  work that buys nothing here. Rust was a close second to Go but adds ownership/borrowing
  ceremony with no corresponding benefit for I/O-bound glue code.
- **VAD: hand-rolled energy-threshold heuristic**, not Silero, for v1. No Swift/Go ML
  ecosystem is mature enough here to justify the integration cost. Upgrade only if
  segmentation quality turns out to be the actual bottleneck — verify before adding
  complexity.
- **ASR runtime: `qwen-asr`** (https://github.com/antirez/qwen-asr), a pure-C implementation
  with `--stream` and `--prompt` (glossary biasing) already built in. Consumed as a
  subprocess, one instance per track. Build is trivial on macOS: no dependency install
  needed beyond the OS's built-in Accelerate framework (`make blas`). No prebuilt binaries
  are published, so it's vendored as a **git submodule** (`third_party/qwen-asr`, pinned to a
  specific commit) and built by `scripts/build-dist.sh` — this exists specifically so
  `transcriptor-ui` (the planned native app, see "Related repos") can produce a fully
  self-contained, one-click-install `.app` with no separate manual clone/build step for the
  end user; models are the only thing still fetched at runtime, deliberately (multi-GB,
  doesn't belong baked into a build). Verified empirically: `--stream` does not print
  incremental partial text to stdout for a single bounded stdin stream — only the final
  transcription, once done. Our own VAD segment boundaries are what stand in for "live"
  granularity, not qwen-asr's internal streaming. One subprocess call per VAD segment
  (measured ~0.15-0.4x realtime including process startup and model load, on both the 0.6B
  and 1.7B models) is fast enough — no need for a persistent daemon process.
- **LLM reread runtime: `llama.cpp`**, installed via `brew install llama.cpp` (prebuilt,
  official Homebrew formula — no source build needed). Model: small, 4-bit quantized,
  sized for the 24GB target machine, not the dev machine. Not wired in yet (deferred).
- **Live display: SSE (Server-Sent Events)**, not WebSocket. The data flow is one-directional
  (server → clients); SSE is plain HTTP, trivially consumable by a terminal client and later
  by a browser (`EventSource`, no client library needed). WebSocket would be
  over-engineering for a need that has no bidirectional requirement today.
- **No Docker.** Ruled out on technical grounds, not preference: Docker Desktop on macOS runs
  a Linux VM with no Metal GPU passthrough, which breaks any Metal-accelerated inference, and
  Core Audio access from inside a container is impractical.

### Message format (SSE)

One JSON object per transcript segment (defined in `internal/transcript`):
```json
{"track": "system", "text": "...", "is_final": false, "timestamp": "2026-08-07T14:32:01.123Z"}
```
`is_final: false` = a streaming ASR partial hypothesis that may still be revised.
`is_final: true` = the segment is closed (VAD ended the utterance).

No timestamp-based reordering across tracks yet — messages are broadcast in the order
`transcribeTrack` produces them. Measured ASR latency is small relative to segment spacing
(3-5s), so this hasn't been observed to cause visible out-of-order delivery. Add a small
reordering buffer in `internal/transcript` if it ever becomes a real problem — not before.

## Related repos

`../audiotee` — the Swift CLI this project spawns as a subprocess for audio capture (see
`internal/capture`). Consumed as a subprocess deliberately, not imported as a Swift library
(`AudioTeeCore` is exposed as one, but that only matters for a Swift consumer, which this
isn't) — see its own `CONTEXT.md` §6.2 for why that repo's TCC-identity signing work matters
specifically because of this. Two tracks (system audio + mic) go to two separate outputs
(stdout + a FIFO), never multiplexed onto one stream — two concurrent Core Audio IO threads
writing raw PCM chunks to a shared fd risked interleaving/corruption.

`../transcriptor-ui` — a native macOS SwiftUI app (separate Xcode project), the "one-click"
way to actually use this — **built and working as of 2026-08-08**, not just planned anymore.
It spawns `transcriptor-ai` as a subprocess (same pattern as `transcriptor-ai` → `audiotee` —
each layer manages the one below it via a subprocess + a clean protocol, never merged into
one project/toolchain) and is an SSE client of it, same as `transcript-tail`, just with a
native UI instead of a terminal. Its Xcode build phase builds this repo's `scripts/build-dist.sh`
and `../audiotee`'s `scripts/build-signed.sh` and embeds all three binaries in the `.app`
bundle — verified fully self-contained (tested with `~/bin/audiotee` moved aside, capture
still worked from the embedded copy). Deliberately **not** a monorepo with this one — see
"Why these choices": mixing Go and Swift tooling in one Xcode project would undermine the
reason Go was chosen for this repo in the first place. transcriptor-ui will eventually let
the user pick ASR model/options and configure the glossary and post-processing
(question-detection, meeting-summary generation via a local LLM) — model selection is still a
dev-time hardcoded path there, nothing else is wired up yet; check transcriptor-ui's own
CLAUDE.md for its current state rather than assuming.

**`-audiotee-binary` flag** exists specifically for transcriptor-ui's benefit: without it,
`internal/capture` only finds `audiotee` via `PATH` then `~/bin/audiotee`, which is fine for
CLI use but wrong for a bundled app that must not depend on anything pre-installed outside
its own `.app` — transcriptor-ui passes its embedded copy's path explicitly.

## Conventions

- Go idioms: `gofmt`/`go vet` clean, standard project layout (`cmd/`, `internal/`).
- Prefer the standard library over third-party Go modules where it's a reasonably close fit
  (`os/exec`, `net/http`, `encoding/json`) — the "self-contained binary, no dependency bazar"
  property is a deliberate, stated goal of this project, not an accident.
- Code and comments in English. Conversation with the user in French (matches the sibling
  `audiotee` repo's convention).
- Don't reintroduce Python, Swift, Docker, or WebSocket for this project without a new,
  explicit reason — see "Why these choices" above.
- User-facing install/usage docs belong in `README.md`, not here — keep this file about
  reasoning and internal state, not instructions for running the tool.

## Status

Capture → VAD → ASR → SSE is implemented and verified end to end:
- `internal/capture` — audiotee subprocess wrapper (stdout + mic FIFO).
- `internal/vad` — energy-threshold segmentation.
- `internal/asr` — one qwen_asr subprocess call per segment.
- `internal/transcript` — `Broadcaster` (SSE), `FileWriter` (durable text file, one line per
  message, fsync'd on every write), `Message` format shared by both.
- `cmd/transcriptor-ai` — wires all of the above together; currently also the test harness.
- `cmd/transcript-tail` — minimal SSE terminal client. Exits calmly (code 0, "stream ended
  (server stopped?)") when the server shuts down, instead of an alarming raw connection error.
- `scripts/build-dist.sh` — builds `qwen_asr` (from the `third_party/qwen-asr` submodule) and
  `transcriptor-ai` into `dist/`, ready for `transcriptor-ui` to embed. Verified: builds
  cleanly from the submodule, and the resulting `dist/qwen_asr` actually transcribes
  (smoke-tested), not just "it compiled."
- `-capture-system`/`-capture-mic` flags (transcriptor-ui's benefit: a UI toggle for what to
  capture). **Asymmetric by necessity, document this if it ever confuses someone**:
  `-capture-mic=false` skips mic capture at the `audiotee` level entirely (no
  `--capture-mic` passed, no mic permission requested) — `internal/capture.Config.CaptureMic`
  controls this. `-capture-system=false` **cannot** do the equivalent — audiotee has no flag
  to skip its system tap, it's always running — so system-only-disabled just means
  `transcriptor-ai` receives system chunks (VAD still runs on them, negligible cost) but never
  dispatches them to an ASR worker (`main.go`'s `dispatch` helper drops them). Verified all
  three modes (system-only, mic-only, both) via CLI: mic-only correctly showed zero
  `track: system` messages even with system audio playing, confirming the drop works, not
  just the flag parsing.

Decision (2026-08-08): the future native macOS UI (`transcriptor-ui`, a separate repo/Xcode
project — see below) will *not* write the transcript file itself; `transcriptor-ai` does, via
`-transcript-file`, specifically because it's the more crash-resistant of the two (simpler
control flow than a UI process) and the whole point of this file is surviving a crash.

Verified: real human speech (English + French code-switching) through an actual microphone,
with both qwen3-asr-0.6b (fast iteration) and qwen3-asr-1.7b (the target model). 1.7B
measured at ~0.4x realtime (2.3s to process a 5.9s clip, model load included) — comfortably
fast enough for one-subprocess-per-segment. Switching models is a runtime flag
(`-asr-model-dir`), not a rebuild.

Three concurrency/signal-handling bugs were found and fixed via real testing (not just code
review):
- `transcribeWorker` (then still per-track, named `transcribeTrack`) was using the
  SIGINT-canceled context for still-queued transcriptions
  (fixed with a separate background context for ASR calls).
- `main()` wasn't waiting for the ASR worker goroutine(s) to finish before exiting (fixed
  with a `sync.WaitGroup`).
- `internal/capture.Stop()` called `cmd.Wait()` before the pipe-reading goroutines had
  finished draining, which `os/exec` docs call out as incorrect and can truncate the last bit
  of audio.
- **`qwen_asr` subprocesses were killed by the terminal's own Ctrl+C**, independent of any of
  the above: child processes inherit the parent's process group by default, and a real
  terminal's Ctrl+C sends SIGINT to the whole foreground process group — not just to
  `transcriptor-ai`. So an in-flight `qwen_asr` call died with "signal: interrupt" regardless
  of the context-handling fix above, which only guards against *our own code* cancelling it.
  Fixed in `internal/asr/runner.go` with `SysProcAttr.Setpgid: true`, putting each `qwen_asr`
  in its own process group. **This bug was invisible to `kill -INT <transcriptor-ai-pid>`**,
  the way earlier manual tests in this session were done — that only signals one PID, not a
  process group, so it never reproduced the real terminal behavior. Verified instead by
  sending SIGINT to the process group (`os.killpg` from a small Python harness, or
  `kill -INT -<pgid>` — careful with the latter run from an interactive shell, it'll also hit
  the shell itself if unrelated jobs share its group).

- **Duplicate text at force-cut segment boundaries**, found from real usage on a real
  meeting-like recording (documentary audio) — visibly bad enough that the transcript was
  unusable, e.g. "à vivre" transcribed once at the end of a segment and again at the start of
  the next. Root cause: the 300ms `Overlap` audio carried into the next force-cut segment (for
  ASR context continuity) was being fully re-transcribed *and* re-published, so both copies
  showed up in the live transcript. Fixed by defaulting `vad.Config.Overlap` to `0` — accept
  an occasional word cut in half at a hard boundary rather than systematic duplication, which
  is far worse for actually reading the transcript. Found and fixed alongside it: `tail()` in
  `internal/vad/vad.go` treated `n <= 0` as "return the whole buffer" instead of empty, which
  would have made `Overlap: 0` carry over the *entire* previous segment instead of nothing —
  fixed to return `nil` for `n <= 0`. If overlap is ever reintroduced, it needs text-level
  dedup on the published output, not just raw audio carry-over.
- **System and mic tracks transcribing the same content** — not a code bug, a structural
  limit of tapping raw Core Audio input: if audio plays through speakers (not headphones),
  the mic picks up the acoustic leakage and transcribes it too. Conferencing apps do their own
  acoustic echo cancellation on the signal they send to other participants, but that doesn't
  touch what our raw mic tap sees. Real-world implication: headphones are likely a hard
  requirement for the two-track "me vs them" diarization to actually hold during real
  meetings, not just a nice-to-have — unresolved, needs a decision (require headphones, or
  build real AEC — a much bigger feature) rather than a quick fix.
- **Shutdown could feel unresponsive** during a large in-flight transcription backlog (no
  feedback while waiting, especially on the slower 1.7B model). Fixed with a "shutting down,
  waiting for in-flight transcriptions..." message and a second Ctrl+C now force-exits
  immediately instead of making the user wait through the full drain.

**ASR memory: single shared worker instead of one per track (2026-08-10).** The 1.7B model
("large" in `transcriptor-ui`) has no quantized-weights option — `qwen_asr` only supports
BF16 (see `third_party/qwen-asr/README.md`, "Memory Requirements": ~6.9GiB static footprint
for the 1.7B model). Before this fix, `cmd/transcriptor-ai/main.go` ran one `transcribeTrack`
goroutine per track (system, mic), each independently spawning `qwen_asr` subprocesses — so
whenever both tracks had speech at the same time, two 1.7B instances ran concurrently,
measured peaking at ~13GiB combined, over half of the 24GB target machine's unified memory
(see "Target hardware"). Fixed by merging both tracks onto one shared worker/channel
(`asrSegs`, `transcribeWorker` — renamed from `transcribeTrack` since it's no longer
per-track): `dispatch()` now sends every enabled track's segments onto the same channel
instead of two separate ones, so `qwen_asr` invocations are strictly serialized regardless of
which track they came from. Per-track chronological order is preserved (VAD's own output
channel is already chronological across both tracks; the worker just drains it in that
order). Trade-off: if both tracks talk at once, one track's transcription now waits for the
other's to finish instead of running in parallel — accepted, since the memory pressure was
the more pressing constraint on the 24GB target machine. Verified empirically, not just "it
compiles": ran the real pipeline with both tracks capturing simultaneous audio (a video
through system output, `say` for spoken input) and polled `pgrep -x qwen_asr` at 0.3s
intervals throughout — max concurrent `qwen_asr` processes observed was 1, confirmed never 2,
while both tracks still produced correctly-ordered transcript segments.

Not yet implemented: post-processing (deferred, see "Out of scope for now"), and anything
from `audiotee`'s CONTEXT.md §6.4-adjacent concerns beyond what's listed here.

**Resume point (2026-08-07):** the live pipeline is done and validated on real speech with
both models, and the force-cut duplication bug is fixed and verified. The system/mic acoustic
leakage issue (headphones vs AEC) is still an open decision. No specific next task was chosen
before pausing beyond that — ask the user
rather than assuming which.