initial commit: live meeting transcription pipeline
Go orchestrator: spawns audiotee for capture (system audio + mic, two tracks), segments with an energy-threshold VAD, transcribes each segment via qwen-asr (subprocess per segment, vendored as a pinned git submodule in third_party/qwen-asr), and serves the live transcript over SSE while also writing it durably to a text file. Includes a glossary/prompt-leakage guard (internal/asr/leak.go) that discards a segment if the model echoes the biasing prompt instead of transcribing. cmd/transcriptor-ai is the main orchestrator; cmd/transcript-tail is a minimal terminal SSE client. See CLAUDE.md for the full architecture and the reasoning behind each choice (Go over Python/Swift/Rust, SSE over WebSocket, why the VAD is a hand-rolled heuristic, etc). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,252 @@
|
||||
# transcriptor-ai
|
||||
|
||||
Local, no-cloud meeting transcription for macOS. See `README.md` for what this is, how to
|
||||
install dependencies, and how to run it — this file is for working on the code: decisions,
|
||||
reasoning, conventions, and current implementation status.
|
||||
|
||||
Design/decision history lives in a long chat transcript, not in this repo yet. This file is
|
||||
the distilled result. If something here seems arbitrary, assume there was a reason discussed
|
||||
elsewhere rather than re-litigating it from scratch — but flag it if it looks wrong given the
|
||||
current code.
|
||||
|
||||
## Goal
|
||||
|
||||
Primary: follow a meeting's transcript live while it happens, so the user can stay focused on
|
||||
listening instead of note-taking. Meetings mix French and English mid-sentence
|
||||
(code-switching), include non-native English accents, and technical/internal jargon that
|
||||
should be correctable via a small per-meeting glossary.
|
||||
|
||||
**Out of scope for now, do not pull forward without being asked:**
|
||||
- Real-time content analysis / suggested questions during the meeting (explicitly phase 2).
|
||||
- Post-processing (fuzzy-match glossary correction, LLM reread via llama.cpp). The pipeline
|
||||
currently publishes qwen-asr's raw output straight to the SSE broadcaster. Revisit only
|
||||
when asked — don't propose implementing this preemptively, it's been deferred twice.
|
||||
|
||||
Only live transcription (capture → VAD → ASR → SSE) is in scope today.
|
||||
|
||||
## Target hardware
|
||||
|
||||
**MacBook (base) M3, 24 GB unified memory.** Not a high-end Max/Pro chip — size and quantize
|
||||
models accordingly (prefer 4-bit; avoid anything like an 8B+ reread model). Do not assume
|
||||
generous headroom. A separate M4 Max/128GB machine exists for development but is not the
|
||||
deployment target; don't let it justify heavier defaults.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
audiotee (subprocess, 2 outputs)
|
||||
├─ stdout → system audio track (PCM)
|
||||
└─ --mic-output → mic track (PCM, via a FIFO for live streaming, not a plain file)
|
||||
│
|
||||
▼ (per track, independently)
|
||||
VAD (simple energy-threshold heuristic, hand-rolled, no ML dependency for v1)
|
||||
│ chunks with overlap
|
||||
▼
|
||||
ASR: qwen-asr subprocess (Qwen3-ASR-1.7B, --stream, --prompt for glossary biasing)
|
||||
│ timestamped segments: {track, text, is_final, timestamp}
|
||||
▼
|
||||
merge (chronological, across both tracks)
|
||||
│
|
||||
▼
|
||||
SSE endpoint (served by this Go binary) — terminal client is the first consumer
|
||||
```
|
||||
|
||||
Post-processing (fuzzy-match glossary correction, then LLM reread via llama.cpp subprocess)
|
||||
was in the original target architecture between merge and SSE, but is explicitly deferred —
|
||||
see "Out of scope for now" above. Raw qwen-asr output is published as-is today.
|
||||
|
||||
### Why these choices (don't re-decide without reason)
|
||||
|
||||
- **Two tracks, two independent VAD+ASR pipelines** (one per track), not one merged audio
|
||||
stream. This is a cheap "me vs them" diarization proxy — no real diarization model needed.
|
||||
Rejected merging into a single flow: it was considered and reverted, keep 2+2.
|
||||
- **Go**, not Python or Swift, for this orchestrator. All the actual ML work (ASR, LLM
|
||||
reread) is delegated to external subprocess binaries regardless of orchestrator language,
|
||||
so the orchestrator's job is just: spawn subprocesses, pipe bytes, run a small HTTP/SSE
|
||||
server, merge by timestamp. That's Go's home turf (`os/exec`, `net/http` stdlib, single
|
||||
static binary, no runtime dependency). Python was rejected for its dependency/environment
|
||||
management overhead; Swift was rejected because its ML ecosystem (Silero VAD, MLX model
|
||||
loading for this specific model) is immature enough to require significant custom porting
|
||||
work that buys nothing here. Rust was a close second to Go but adds ownership/borrowing
|
||||
ceremony with no corresponding benefit for I/O-bound glue code.
|
||||
- **VAD: hand-rolled energy-threshold heuristic**, not Silero, for v1. No Swift/Go ML
|
||||
ecosystem is mature enough here to justify the integration cost. Upgrade only if
|
||||
segmentation quality turns out to be the actual bottleneck — verify before adding
|
||||
complexity.
|
||||
- **ASR runtime: `qwen-asr`** (https://github.com/antirez/qwen-asr), a pure-C implementation
|
||||
with `--stream` and `--prompt` (glossary biasing) already built in. Consumed as a
|
||||
subprocess, one instance per track. Build is trivial on macOS: no dependency install
|
||||
needed beyond the OS's built-in Accelerate framework (`make blas`). No prebuilt binaries
|
||||
are published, so it's vendored as a **git submodule** (`third_party/qwen-asr`, pinned to a
|
||||
specific commit) and built by `scripts/build-dist.sh` — this exists specifically so
|
||||
`transcriptor-ui` (the planned native app, see "Related repos") can produce a fully
|
||||
self-contained, one-click-install `.app` with no separate manual clone/build step for the
|
||||
end user; models are the only thing still fetched at runtime, deliberately (multi-GB,
|
||||
doesn't belong baked into a build). Verified empirically: `--stream` does not print
|
||||
incremental partial text to stdout for a single bounded stdin stream — only the final
|
||||
transcription, once done. Our own VAD segment boundaries are what stand in for "live"
|
||||
granularity, not qwen-asr's internal streaming. One subprocess call per VAD segment
|
||||
(measured ~0.15-0.4x realtime including process startup and model load, on both the 0.6B
|
||||
and 1.7B models) is fast enough — no need for a persistent daemon process.
|
||||
- **LLM reread runtime: `llama.cpp`**, installed via `brew install llama.cpp` (prebuilt,
|
||||
official Homebrew formula — no source build needed). Model: small, 4-bit quantized,
|
||||
sized for the 24GB target machine, not the dev machine. Not wired in yet (deferred).
|
||||
- **Live display: SSE (Server-Sent Events)**, not WebSocket. The data flow is one-directional
|
||||
(server → clients); SSE is plain HTTP, trivially consumable by a terminal client and later
|
||||
by a browser (`EventSource`, no client library needed). WebSocket would be
|
||||
over-engineering for a need that has no bidirectional requirement today.
|
||||
- **No Docker.** Ruled out on technical grounds, not preference: Docker Desktop on macOS runs
|
||||
a Linux VM with no Metal GPU passthrough, which breaks any Metal-accelerated inference, and
|
||||
Core Audio access from inside a container is impractical.
|
||||
|
||||
### Message format (SSE)
|
||||
|
||||
One JSON object per transcript segment (defined in `internal/transcript`):
|
||||
```json
|
||||
{"track": "system", "text": "...", "is_final": false, "timestamp": "2026-08-07T14:32:01.123Z"}
|
||||
```
|
||||
`is_final: false` = a streaming ASR partial hypothesis that may still be revised.
|
||||
`is_final: true` = the segment is closed (VAD ended the utterance).
|
||||
|
||||
No timestamp-based reordering across tracks yet — messages are broadcast in the order
|
||||
`transcribeTrack` produces them. Measured ASR latency is small relative to segment spacing
|
||||
(3-5s), so this hasn't been observed to cause visible out-of-order delivery. Add a small
|
||||
reordering buffer in `internal/transcript` if it ever becomes a real problem — not before.
|
||||
|
||||
## Related repos
|
||||
|
||||
`../audiotee` — the Swift CLI this project spawns as a subprocess for audio capture (see
|
||||
`internal/capture`). Consumed as a subprocess deliberately, not imported as a Swift library
|
||||
(`AudioTeeCore` is exposed as one, but that only matters for a Swift consumer, which this
|
||||
isn't) — see its own `CONTEXT.md` §6.2 for why that repo's TCC-identity signing work matters
|
||||
specifically because of this. Two tracks (system audio + mic) go to two separate outputs
|
||||
(stdout + a FIFO), never multiplexed onto one stream — two concurrent Core Audio IO threads
|
||||
writing raw PCM chunks to a shared fd risked interleaving/corruption.
|
||||
|
||||
`../transcriptor-ui` — a native macOS SwiftUI app (separate Xcode project), the "one-click"
|
||||
way to actually use this — **built and working as of 2026-08-08**, not just planned anymore.
|
||||
It spawns `transcriptor-ai` as a subprocess (same pattern as `transcriptor-ai` → `audiotee` —
|
||||
each layer manages the one below it via a subprocess + a clean protocol, never merged into
|
||||
one project/toolchain) and is an SSE client of it, same as `transcript-tail`, just with a
|
||||
native UI instead of a terminal. Its Xcode build phase builds this repo's `scripts/build-dist.sh`
|
||||
and `../audiotee`'s `scripts/build-signed.sh` and embeds all three binaries in the `.app`
|
||||
bundle — verified fully self-contained (tested with `~/bin/audiotee` moved aside, capture
|
||||
still worked from the embedded copy). Deliberately **not** a monorepo with this one — see
|
||||
"Why these choices": mixing Go and Swift tooling in one Xcode project would undermine the
|
||||
reason Go was chosen for this repo in the first place. transcriptor-ui will eventually let
|
||||
the user pick ASR model/options and configure the glossary and post-processing
|
||||
(question-detection, meeting-summary generation via a local LLM) — model selection is still a
|
||||
dev-time hardcoded path there, nothing else is wired up yet; check transcriptor-ui's own
|
||||
CLAUDE.md for its current state rather than assuming.
|
||||
|
||||
**`-audiotee-binary` flag** exists specifically for transcriptor-ui's benefit: without it,
|
||||
`internal/capture` only finds `audiotee` via `PATH` then `~/bin/audiotee`, which is fine for
|
||||
CLI use but wrong for a bundled app that must not depend on anything pre-installed outside
|
||||
its own `.app` — transcriptor-ui passes its embedded copy's path explicitly.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Go idioms: `gofmt`/`go vet` clean, standard project layout (`cmd/`, `internal/`).
|
||||
- Prefer the standard library over third-party Go modules where it's a reasonably close fit
|
||||
(`os/exec`, `net/http`, `encoding/json`) — the "self-contained binary, no dependency bazar"
|
||||
property is a deliberate, stated goal of this project, not an accident.
|
||||
- Code and comments in English. Conversation with the user in French (matches the sibling
|
||||
`audiotee` repo's convention).
|
||||
- Don't reintroduce Python, Swift, Docker, or WebSocket for this project without a new,
|
||||
explicit reason — see "Why these choices" above.
|
||||
- User-facing install/usage docs belong in `README.md`, not here — keep this file about
|
||||
reasoning and internal state, not instructions for running the tool.
|
||||
|
||||
## Status
|
||||
|
||||
Capture → VAD → ASR → SSE is implemented and verified end to end:
|
||||
- `internal/capture` — audiotee subprocess wrapper (stdout + mic FIFO).
|
||||
- `internal/vad` — energy-threshold segmentation.
|
||||
- `internal/asr` — one qwen_asr subprocess call per segment.
|
||||
- `internal/transcript` — `Broadcaster` (SSE), `FileWriter` (durable text file, one line per
|
||||
message, fsync'd on every write), `Message` format shared by both.
|
||||
- `cmd/transcriptor-ai` — wires all of the above together; currently also the test harness.
|
||||
- `cmd/transcript-tail` — minimal SSE terminal client. Exits calmly (code 0, "stream ended
|
||||
(server stopped?)") when the server shuts down, instead of an alarming raw connection error.
|
||||
- `scripts/build-dist.sh` — builds `qwen_asr` (from the `third_party/qwen-asr` submodule) and
|
||||
`transcriptor-ai` into `dist/`, ready for `transcriptor-ui` to embed. Verified: builds
|
||||
cleanly from the submodule, and the resulting `dist/qwen_asr` actually transcribes
|
||||
(smoke-tested), not just "it compiled."
|
||||
- `-capture-system`/`-capture-mic` flags (transcriptor-ui's benefit: a UI toggle for what to
|
||||
capture). **Asymmetric by necessity, document this if it ever confuses someone**:
|
||||
`-capture-mic=false` skips mic capture at the `audiotee` level entirely (no
|
||||
`--capture-mic` passed, no mic permission requested) — `internal/capture.Config.CaptureMic`
|
||||
controls this. `-capture-system=false` **cannot** do the equivalent — audiotee has no flag
|
||||
to skip its system tap, it's always running — so system-only-disabled just means
|
||||
`transcriptor-ai` receives system chunks (VAD still runs on them, negligible cost) but never
|
||||
dispatches them to an ASR worker (`main.go`'s `dispatch` helper drops them). Verified all
|
||||
three modes (system-only, mic-only, both) via CLI: mic-only correctly showed zero
|
||||
`track: system` messages even with system audio playing, confirming the drop works, not
|
||||
just the flag parsing.
|
||||
|
||||
Decision (2026-08-08): the future native macOS UI (`transcriptor-ui`, a separate repo/Xcode
|
||||
project — see below) will *not* write the transcript file itself; `transcriptor-ai` does, via
|
||||
`-transcript-file`, specifically because it's the more crash-resistant of the two (simpler
|
||||
control flow than a UI process) and the whole point of this file is surviving a crash.
|
||||
|
||||
Verified: real human speech (English + French code-switching) through an actual microphone,
|
||||
with both qwen3-asr-0.6b (fast iteration) and qwen3-asr-1.7b (the target model). 1.7B
|
||||
measured at ~0.4x realtime (2.3s to process a 5.9s clip, model load included) — comfortably
|
||||
fast enough for one-subprocess-per-segment. Switching models is a runtime flag
|
||||
(`-asr-model-dir`), not a rebuild.
|
||||
|
||||
Three concurrency/signal-handling bugs were found and fixed via real testing (not just code
|
||||
review):
|
||||
- `transcribeTrack` was using the SIGINT-canceled context for still-queued transcriptions
|
||||
(fixed with a separate background context for ASR calls).
|
||||
- `main()` wasn't waiting for `transcribeTrack` goroutines to finish before exiting (fixed
|
||||
with a `sync.WaitGroup`).
|
||||
- `internal/capture.Stop()` called `cmd.Wait()` before the pipe-reading goroutines had
|
||||
finished draining, which `os/exec` docs call out as incorrect and can truncate the last bit
|
||||
of audio.
|
||||
- **`qwen_asr` subprocesses were killed by the terminal's own Ctrl+C**, independent of any of
|
||||
the above: child processes inherit the parent's process group by default, and a real
|
||||
terminal's Ctrl+C sends SIGINT to the whole foreground process group — not just to
|
||||
`transcriptor-ai`. So an in-flight `qwen_asr` call died with "signal: interrupt" regardless
|
||||
of the context-handling fix above, which only guards against *our own code* cancelling it.
|
||||
Fixed in `internal/asr/runner.go` with `SysProcAttr.Setpgid: true`, putting each `qwen_asr`
|
||||
in its own process group. **This bug was invisible to `kill -INT <transcriptor-ai-pid>`**,
|
||||
the way earlier manual tests in this session were done — that only signals one PID, not a
|
||||
process group, so it never reproduced the real terminal behavior. Verified instead by
|
||||
sending SIGINT to the process group (`os.killpg` from a small Python harness, or
|
||||
`kill -INT -<pgid>` — careful with the latter run from an interactive shell, it'll also hit
|
||||
the shell itself if unrelated jobs share its group).
|
||||
|
||||
- **Duplicate text at force-cut segment boundaries**, found from real usage on a real
|
||||
meeting-like recording (documentary audio) — visibly bad enough that the transcript was
|
||||
unusable, e.g. "à vivre" transcribed once at the end of a segment and again at the start of
|
||||
the next. Root cause: the 300ms `Overlap` audio carried into the next force-cut segment (for
|
||||
ASR context continuity) was being fully re-transcribed *and* re-published, so both copies
|
||||
showed up in the live transcript. Fixed by defaulting `vad.Config.Overlap` to `0` — accept
|
||||
an occasional word cut in half at a hard boundary rather than systematic duplication, which
|
||||
is far worse for actually reading the transcript. Found and fixed alongside it: `tail()` in
|
||||
`internal/vad/vad.go` treated `n <= 0` as "return the whole buffer" instead of empty, which
|
||||
would have made `Overlap: 0` carry over the *entire* previous segment instead of nothing —
|
||||
fixed to return `nil` for `n <= 0`. If overlap is ever reintroduced, it needs text-level
|
||||
dedup on the published output, not just raw audio carry-over.
|
||||
- **System and mic tracks transcribing the same content** — not a code bug, a structural
|
||||
limit of tapping raw Core Audio input: if audio plays through speakers (not headphones),
|
||||
the mic picks up the acoustic leakage and transcribes it too. Conferencing apps do their own
|
||||
acoustic echo cancellation on the signal they send to other participants, but that doesn't
|
||||
touch what our raw mic tap sees. Real-world implication: headphones are likely a hard
|
||||
requirement for the two-track "me vs them" diarization to actually hold during real
|
||||
meetings, not just a nice-to-have — unresolved, needs a decision (require headphones, or
|
||||
build real AEC — a much bigger feature) rather than a quick fix.
|
||||
- **Shutdown could feel unresponsive** during a large in-flight transcription backlog (no
|
||||
feedback while waiting, especially on the slower 1.7B model). Fixed with a "shutting down,
|
||||
waiting for in-flight transcriptions..." message and a second Ctrl+C now force-exits
|
||||
immediately instead of making the user wait through the full drain.
|
||||
|
||||
Not yet implemented: post-processing (deferred, see "Out of scope for now"), and anything
|
||||
from `audiotee`'s CONTEXT.md §6.4-adjacent concerns beyond what's listed here.
|
||||
|
||||
**Resume point (2026-08-07):** the live pipeline is done and validated on real speech with
|
||||
both models, and the force-cut duplication bug is fixed and verified. The system/mic acoustic
|
||||
leakage issue (headphones vs AEC) is still an open decision. No specific next task was chosen
|
||||
before pausing beyond that — ask the user
|
||||
rather than assuming which.
|
||||
Reference in New Issue
Block a user