commit 3525e6b8fbe1222d56c974c60a80a3fc3d0f87d4 Author: sttlab-tech Date: Sun Aug 9 13:18:57 2026 +0200 initial commit: live meeting transcription pipeline Go orchestrator: spawns audiotee for capture (system audio + mic, two tracks), segments with an energy-threshold VAD, transcribes each segment via qwen-asr (subprocess per segment, vendored as a pinned git submodule in third_party/qwen-asr), and serves the live transcript over SSE while also writing it durably to a text file. Includes a glossary/prompt-leakage guard (internal/asr/leak.go) that discards a segment if the model echoes the biasing prompt instead of transcribing. cmd/transcriptor-ai is the main orchestrator; cmd/transcript-tail is a minimal terminal SSE client. See CLAUDE.md for the full architecture and the reasoning behind each choice (Go over Python/Swift/Rust, SSE over WebSocket, why the VAD is a hand-rolled heuristic, etc). Co-Authored-By: Claude Sonnet 5 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d1bd9ca --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +.DS_Store +/transcriptor-ai +/transcript-tail +/dist +*.pcm +*.wav diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..3ad7d4b --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "third_party/qwen-asr"] + path = third_party/qwen-asr + url = https://github.com/antirez/qwen-asr diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2b2ad4a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,252 @@ +# transcriptor-ai + +Local, no-cloud meeting transcription for macOS. See `README.md` for what this is, how to +install dependencies, and how to run it — this file is for working on the code: decisions, +reasoning, conventions, and current implementation status. + +Design/decision history lives in a long chat transcript, not in this repo yet. This file is +the distilled result. If something here seems arbitrary, assume there was a reason discussed +elsewhere rather than re-litigating it from scratch — but flag it if it looks wrong given the +current code. + +## Goal + +Primary: follow a meeting's transcript live while it happens, so the user can stay focused on +listening instead of note-taking. Meetings mix French and English mid-sentence +(code-switching), include non-native English accents, and technical/internal jargon that +should be correctable via a small per-meeting glossary. + +**Out of scope for now, do not pull forward without being asked:** +- Real-time content analysis / suggested questions during the meeting (explicitly phase 2). +- Post-processing (fuzzy-match glossary correction, LLM reread via llama.cpp). The pipeline + currently publishes qwen-asr's raw output straight to the SSE broadcaster. Revisit only + when asked — don't propose implementing this preemptively, it's been deferred twice. + +Only live transcription (capture → VAD → ASR → SSE) is in scope today. + +## Target hardware + +**MacBook (base) M3, 24 GB unified memory.** Not a high-end Max/Pro chip — size and quantize +models accordingly (prefer 4-bit; avoid anything like an 8B+ reread model). Do not assume +generous headroom. A separate M4 Max/128GB machine exists for development but is not the +deployment target; don't let it justify heavier defaults. + +## Architecture + +``` +audiotee (subprocess, 2 outputs) + ├─ stdout → system audio track (PCM) + └─ --mic-output → mic track (PCM, via a FIFO for live streaming, not a plain file) + │ + ▼ (per track, independently) + VAD (simple energy-threshold heuristic, hand-rolled, no ML dependency for v1) + │ chunks with overlap + ▼ + ASR: qwen-asr subprocess (Qwen3-ASR-1.7B, --stream, --prompt for glossary biasing) + │ timestamped segments: {track, text, is_final, timestamp} + ▼ + merge (chronological, across both tracks) + │ + ▼ + SSE endpoint (served by this Go binary) — terminal client is the first consumer +``` + +Post-processing (fuzzy-match glossary correction, then LLM reread via llama.cpp subprocess) +was in the original target architecture between merge and SSE, but is explicitly deferred — +see "Out of scope for now" above. Raw qwen-asr output is published as-is today. + +### Why these choices (don't re-decide without reason) + +- **Two tracks, two independent VAD+ASR pipelines** (one per track), not one merged audio + stream. This is a cheap "me vs them" diarization proxy — no real diarization model needed. + Rejected merging into a single flow: it was considered and reverted, keep 2+2. +- **Go**, not Python or Swift, for this orchestrator. All the actual ML work (ASR, LLM + reread) is delegated to external subprocess binaries regardless of orchestrator language, + so the orchestrator's job is just: spawn subprocesses, pipe bytes, run a small HTTP/SSE + server, merge by timestamp. That's Go's home turf (`os/exec`, `net/http` stdlib, single + static binary, no runtime dependency). Python was rejected for its dependency/environment + management overhead; Swift was rejected because its ML ecosystem (Silero VAD, MLX model + loading for this specific model) is immature enough to require significant custom porting + work that buys nothing here. Rust was a close second to Go but adds ownership/borrowing + ceremony with no corresponding benefit for I/O-bound glue code. +- **VAD: hand-rolled energy-threshold heuristic**, not Silero, for v1. No Swift/Go ML + ecosystem is mature enough here to justify the integration cost. Upgrade only if + segmentation quality turns out to be the actual bottleneck — verify before adding + complexity. +- **ASR runtime: `qwen-asr`** (https://github.com/antirez/qwen-asr), a pure-C implementation + with `--stream` and `--prompt` (glossary biasing) already built in. Consumed as a + subprocess, one instance per track. Build is trivial on macOS: no dependency install + needed beyond the OS's built-in Accelerate framework (`make blas`). No prebuilt binaries + are published, so it's vendored as a **git submodule** (`third_party/qwen-asr`, pinned to a + specific commit) and built by `scripts/build-dist.sh` — this exists specifically so + `transcriptor-ui` (the planned native app, see "Related repos") can produce a fully + self-contained, one-click-install `.app` with no separate manual clone/build step for the + end user; models are the only thing still fetched at runtime, deliberately (multi-GB, + doesn't belong baked into a build). Verified empirically: `--stream` does not print + incremental partial text to stdout for a single bounded stdin stream — only the final + transcription, once done. Our own VAD segment boundaries are what stand in for "live" + granularity, not qwen-asr's internal streaming. One subprocess call per VAD segment + (measured ~0.15-0.4x realtime including process startup and model load, on both the 0.6B + and 1.7B models) is fast enough — no need for a persistent daemon process. +- **LLM reread runtime: `llama.cpp`**, installed via `brew install llama.cpp` (prebuilt, + official Homebrew formula — no source build needed). Model: small, 4-bit quantized, + sized for the 24GB target machine, not the dev machine. Not wired in yet (deferred). +- **Live display: SSE (Server-Sent Events)**, not WebSocket. The data flow is one-directional + (server → clients); SSE is plain HTTP, trivially consumable by a terminal client and later + by a browser (`EventSource`, no client library needed). WebSocket would be + over-engineering for a need that has no bidirectional requirement today. +- **No Docker.** Ruled out on technical grounds, not preference: Docker Desktop on macOS runs + a Linux VM with no Metal GPU passthrough, which breaks any Metal-accelerated inference, and + Core Audio access from inside a container is impractical. + +### Message format (SSE) + +One JSON object per transcript segment (defined in `internal/transcript`): +```json +{"track": "system", "text": "...", "is_final": false, "timestamp": "2026-08-07T14:32:01.123Z"} +``` +`is_final: false` = a streaming ASR partial hypothesis that may still be revised. +`is_final: true` = the segment is closed (VAD ended the utterance). + +No timestamp-based reordering across tracks yet — messages are broadcast in the order +`transcribeTrack` produces them. Measured ASR latency is small relative to segment spacing +(3-5s), so this hasn't been observed to cause visible out-of-order delivery. Add a small +reordering buffer in `internal/transcript` if it ever becomes a real problem — not before. + +## Related repos + +`../audiotee` — the Swift CLI this project spawns as a subprocess for audio capture (see +`internal/capture`). Consumed as a subprocess deliberately, not imported as a Swift library +(`AudioTeeCore` is exposed as one, but that only matters for a Swift consumer, which this +isn't) — see its own `CONTEXT.md` §6.2 for why that repo's TCC-identity signing work matters +specifically because of this. Two tracks (system audio + mic) go to two separate outputs +(stdout + a FIFO), never multiplexed onto one stream — two concurrent Core Audio IO threads +writing raw PCM chunks to a shared fd risked interleaving/corruption. + +`../transcriptor-ui` — a native macOS SwiftUI app (separate Xcode project), the "one-click" +way to actually use this — **built and working as of 2026-08-08**, not just planned anymore. +It spawns `transcriptor-ai` as a subprocess (same pattern as `transcriptor-ai` → `audiotee` — +each layer manages the one below it via a subprocess + a clean protocol, never merged into +one project/toolchain) and is an SSE client of it, same as `transcript-tail`, just with a +native UI instead of a terminal. Its Xcode build phase builds this repo's `scripts/build-dist.sh` +and `../audiotee`'s `scripts/build-signed.sh` and embeds all three binaries in the `.app` +bundle — verified fully self-contained (tested with `~/bin/audiotee` moved aside, capture +still worked from the embedded copy). Deliberately **not** a monorepo with this one — see +"Why these choices": mixing Go and Swift tooling in one Xcode project would undermine the +reason Go was chosen for this repo in the first place. transcriptor-ui will eventually let +the user pick ASR model/options and configure the glossary and post-processing +(question-detection, meeting-summary generation via a local LLM) — model selection is still a +dev-time hardcoded path there, nothing else is wired up yet; check transcriptor-ui's own +CLAUDE.md for its current state rather than assuming. + +**`-audiotee-binary` flag** exists specifically for transcriptor-ui's benefit: without it, +`internal/capture` only finds `audiotee` via `PATH` then `~/bin/audiotee`, which is fine for +CLI use but wrong for a bundled app that must not depend on anything pre-installed outside +its own `.app` — transcriptor-ui passes its embedded copy's path explicitly. + +## Conventions + +- Go idioms: `gofmt`/`go vet` clean, standard project layout (`cmd/`, `internal/`). +- Prefer the standard library over third-party Go modules where it's a reasonably close fit + (`os/exec`, `net/http`, `encoding/json`) — the "self-contained binary, no dependency bazar" + property is a deliberate, stated goal of this project, not an accident. +- Code and comments in English. Conversation with the user in French (matches the sibling + `audiotee` repo's convention). +- Don't reintroduce Python, Swift, Docker, or WebSocket for this project without a new, + explicit reason — see "Why these choices" above. +- User-facing install/usage docs belong in `README.md`, not here — keep this file about + reasoning and internal state, not instructions for running the tool. + +## Status + +Capture → VAD → ASR → SSE is implemented and verified end to end: +- `internal/capture` — audiotee subprocess wrapper (stdout + mic FIFO). +- `internal/vad` — energy-threshold segmentation. +- `internal/asr` — one qwen_asr subprocess call per segment. +- `internal/transcript` — `Broadcaster` (SSE), `FileWriter` (durable text file, one line per + message, fsync'd on every write), `Message` format shared by both. +- `cmd/transcriptor-ai` — wires all of the above together; currently also the test harness. +- `cmd/transcript-tail` — minimal SSE terminal client. Exits calmly (code 0, "stream ended + (server stopped?)") when the server shuts down, instead of an alarming raw connection error. +- `scripts/build-dist.sh` — builds `qwen_asr` (from the `third_party/qwen-asr` submodule) and + `transcriptor-ai` into `dist/`, ready for `transcriptor-ui` to embed. Verified: builds + cleanly from the submodule, and the resulting `dist/qwen_asr` actually transcribes + (smoke-tested), not just "it compiled." +- `-capture-system`/`-capture-mic` flags (transcriptor-ui's benefit: a UI toggle for what to + capture). **Asymmetric by necessity, document this if it ever confuses someone**: + `-capture-mic=false` skips mic capture at the `audiotee` level entirely (no + `--capture-mic` passed, no mic permission requested) — `internal/capture.Config.CaptureMic` + controls this. `-capture-system=false` **cannot** do the equivalent — audiotee has no flag + to skip its system tap, it's always running — so system-only-disabled just means + `transcriptor-ai` receives system chunks (VAD still runs on them, negligible cost) but never + dispatches them to an ASR worker (`main.go`'s `dispatch` helper drops them). Verified all + three modes (system-only, mic-only, both) via CLI: mic-only correctly showed zero + `track: system` messages even with system audio playing, confirming the drop works, not + just the flag parsing. + +Decision (2026-08-08): the future native macOS UI (`transcriptor-ui`, a separate repo/Xcode +project — see below) will *not* write the transcript file itself; `transcriptor-ai` does, via +`-transcript-file`, specifically because it's the more crash-resistant of the two (simpler +control flow than a UI process) and the whole point of this file is surviving a crash. + +Verified: real human speech (English + French code-switching) through an actual microphone, +with both qwen3-asr-0.6b (fast iteration) and qwen3-asr-1.7b (the target model). 1.7B +measured at ~0.4x realtime (2.3s to process a 5.9s clip, model load included) — comfortably +fast enough for one-subprocess-per-segment. Switching models is a runtime flag +(`-asr-model-dir`), not a rebuild. + +Three concurrency/signal-handling bugs were found and fixed via real testing (not just code +review): +- `transcribeTrack` was using the SIGINT-canceled context for still-queued transcriptions + (fixed with a separate background context for ASR calls). +- `main()` wasn't waiting for `transcribeTrack` goroutines to finish before exiting (fixed + with a `sync.WaitGroup`). +- `internal/capture.Stop()` called `cmd.Wait()` before the pipe-reading goroutines had + finished draining, which `os/exec` docs call out as incorrect and can truncate the last bit + of audio. +- **`qwen_asr` subprocesses were killed by the terminal's own Ctrl+C**, independent of any of + the above: child processes inherit the parent's process group by default, and a real + terminal's Ctrl+C sends SIGINT to the whole foreground process group — not just to + `transcriptor-ai`. So an in-flight `qwen_asr` call died with "signal: interrupt" regardless + of the context-handling fix above, which only guards against *our own code* cancelling it. + Fixed in `internal/asr/runner.go` with `SysProcAttr.Setpgid: true`, putting each `qwen_asr` + in its own process group. **This bug was invisible to `kill -INT `**, + the way earlier manual tests in this session were done — that only signals one PID, not a + process group, so it never reproduced the real terminal behavior. Verified instead by + sending SIGINT to the process group (`os.killpg` from a small Python harness, or + `kill -INT -` — careful with the latter run from an interactive shell, it'll also hit + the shell itself if unrelated jobs share its group). + +- **Duplicate text at force-cut segment boundaries**, found from real usage on a real + meeting-like recording (documentary audio) — visibly bad enough that the transcript was + unusable, e.g. "à vivre" transcribed once at the end of a segment and again at the start of + the next. Root cause: the 300ms `Overlap` audio carried into the next force-cut segment (for + ASR context continuity) was being fully re-transcribed *and* re-published, so both copies + showed up in the live transcript. Fixed by defaulting `vad.Config.Overlap` to `0` — accept + an occasional word cut in half at a hard boundary rather than systematic duplication, which + is far worse for actually reading the transcript. Found and fixed alongside it: `tail()` in + `internal/vad/vad.go` treated `n <= 0` as "return the whole buffer" instead of empty, which + would have made `Overlap: 0` carry over the *entire* previous segment instead of nothing — + fixed to return `nil` for `n <= 0`. If overlap is ever reintroduced, it needs text-level + dedup on the published output, not just raw audio carry-over. +- **System and mic tracks transcribing the same content** — not a code bug, a structural + limit of tapping raw Core Audio input: if audio plays through speakers (not headphones), + the mic picks up the acoustic leakage and transcribes it too. Conferencing apps do their own + acoustic echo cancellation on the signal they send to other participants, but that doesn't + touch what our raw mic tap sees. Real-world implication: headphones are likely a hard + requirement for the two-track "me vs them" diarization to actually hold during real + meetings, not just a nice-to-have — unresolved, needs a decision (require headphones, or + build real AEC — a much bigger feature) rather than a quick fix. +- **Shutdown could feel unresponsive** during a large in-flight transcription backlog (no + feedback while waiting, especially on the slower 1.7B model). Fixed with a "shutting down, + waiting for in-flight transcriptions..." message and a second Ctrl+C now force-exits + immediately instead of making the user wait through the full drain. + +Not yet implemented: post-processing (deferred, see "Out of scope for now"), and anything +from `audiotee`'s CONTEXT.md §6.4-adjacent concerns beyond what's listed here. + +**Resume point (2026-08-07):** the live pipeline is done and validated on real speech with +both models, and the force-cut duplication bug is fixed and verified. The system/mic acoustic +leakage issue (headphones vs AEC) is still an open decision. No specific next task was chosen +before pausing beyond that — ask the user +rather than assuming which. diff --git a/README.md b/README.md new file mode 100644 index 0000000..64ab562 --- /dev/null +++ b/README.md @@ -0,0 +1,120 @@ +# transcriptor-ai + +Local, no-cloud meeting transcription for macOS. Captures system audio and microphone as two +separate tracks, transcribes both live, and serves a live-updating transcript over HTTP (SSE). +Everything runs on-device — no external API calls at runtime. + +## How it works + +``` +audiotee (system audio + mic, 2 tracks) + → VAD (speech segmentation, per track) + → qwen-asr (speech-to-text, per track) + → live transcript, served over SSE +``` + +Two tracks (system audio vs microphone) act as a cheap "me vs them" diarization: whatever you +hear vs whatever you say, without needing a real speaker-diarization model. + +## Prerequisites + +- macOS (Apple Silicon) +- Go 1.26+ (`brew install go`) +- **[audiotee](../audiotee)** — built and signed with a stable identity so its audio + permissions survive rebuilds. Follow that repo's own README/CONTEXT.md first; you should + end up with a working binary, e.g. at `~/bin/audiotee`. +- **qwen-asr** — vendored as a git submodule (`third_party/qwen-asr`, pinned to a specific + commit), built automatically by `scripts/build-dist.sh` below. You still need to download a + model yourself (not part of the build — see "Models"), since that's a multi-GB download, + not something to fetch on every build: + ```bash + git submodule update --init third_party/qwen-asr # if you haven't run build-dist.sh yet + cd third_party/qwen-asr && ./download_model.sh --model large # or --model small + ``` + +`llama.cpp` (`brew install llama.cpp`) will be needed once post-processing (glossary +correction, LLM reread) is implemented, but isn't used yet — no need to install it today. + +## Build + +```bash +git clone --recurse-submodules # or: git submodule update --init +scripts/build-dist.sh +``` +Builds `qwen_asr` (from the submodule) and `transcriptor-ai` into `dist/`. Also build +`transcript-tail` directly, it's not part of `dist/` (that's for what `transcriptor-ui`, the +future native app, embeds — see `CLAUDE.md`): +```bash +go build -o transcript-tail ./cmd/transcript-tail +``` + +## Usage + +Start the transcriber: + +```bash +./dist/transcriptor-ai \ + -asr-binary ./dist/qwen_asr \ + -asr-model-dir ./third_party/qwen-asr/qwen3-asr-1.7b \ + -transcript-file ~/Desktop/meeting-transcript.txt +``` + +In another terminal, follow the live transcript: + +```bash +./transcript-tail +``` + +Output looks like: +``` +[mic ] Bonjour, ceci est un test. +[system…] Hello, this is a partial hypothesis that may still be revised +``` +(no marker = a closed/final segment; `…` = a partial hypothesis, ASR-in-progress) + +Stop with Ctrl+C — shutdown is graceful (in-flight transcriptions finish before exiting). + +### Flags + +`transcriptor-ai`: + +| Flag | Default | Description | +|---|---|---| +| `-audiotee-binary` | PATH, then `~/bin/audiotee` | Path to the `audiotee` binary | +| `-asr-binary` | *(required)* | Path to the `qwen_asr` binary | +| `-asr-model-dir` | *(required)* | Path to a qwen-asr model directory | +| `-prompt` | *(empty)* | Glossary biasing text passed to qwen-asr (e.g. `"Preserve spelling: Kubernetes, PostgreSQL"`) | +| `-http-addr` | `:8420` | Address to serve the live transcript SSE endpoint on | +| `-transcript-file` | *(empty)* | Path to append the live transcript to as plain text, flushed to disk after every line. Not written if empty. | +| `-capture-system` | `true` | Transcribe the system audio track. Note: audiotee's system tap always runs regardless of this flag — disabling it only stops *transcribing* that track, it doesn't skip the underlying capture (audiotee has no flag for that). | +| `-capture-mic` | `true` | Capture and transcribe the microphone track. Unlike `-capture-system`, disabling this actually skips mic capture entirely (no `--capture-mic` passed to audiotee, no microphone permission requested). | + +At least one of `-capture-system`/`-capture-mic` must stay enabled. + +`transcript-tail`: + +| Flag | Default | Description | +|---|---|---| +| `-url` | `http://localhost:8420/events` | transcriptor-ai SSE endpoint to follow | + +### Consuming the transcript programmatically + +The SSE endpoint (`http://localhost:8420/events` by default) streams one JSON object per +transcript segment: +```json +{"track": "system", "text": "...", "is_final": false, "timestamp": "2026-08-07T14:32:01.123Z"} +``` +Any SSE client works, e.g. `curl -N http://localhost:8420/events`, or a browser's +`EventSource`. + +## Status + +Live transcription (capture → VAD → ASR → SSE) works end to end, tested with real speech +(English + French, code-switching) through an actual microphone. Not yet implemented: +post-processing (glossary fuzzy-matching, LLM reread) and real-time content analysis — see +`CLAUDE.md` for what's deliberately deferred and why. + +## Related + +- [audiotee](../audiotee) — the system/mic audio capture CLI this project depends on. +- [antirez/qwen-asr](https://github.com/antirez/qwen-asr) — the speech-to-text engine used. diff --git a/cmd/transcript-tail/main.go b/cmd/transcript-tail/main.go new file mode 100644 index 0000000..ea264b4 --- /dev/null +++ b/cmd/transcript-tail/main.go @@ -0,0 +1,66 @@ +// Command transcript-tail is a minimal terminal client for transcriptor-ai's +// live SSE transcript endpoint — connects, parses each `data: {...}` event, +// and prints just the text, instead of raw JSON (`curl -N` works too, but +// isn't pleasant to read live). See CLAUDE.md for the SSE message format. +package main + +import ( + "bufio" + "encoding/json" + "flag" + "fmt" + "net/http" + "os" + "strings" + + "github.com/stephanetailland/transcriptor-ai/internal/transcript" +) + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "transcript-tail:", err) + os.Exit(1) + } +} + +func run() error { + url := flag.String("url", "http://localhost:8420/events", "transcriptor-ai SSE endpoint to follow") + flag.Parse() + + resp, err := http.Get(*url) + if err != nil { + return fmt.Errorf("connect: %w", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("unexpected status: %s", resp.Status) + } + + scanner := bufio.NewScanner(resp.Body) + for scanner.Scan() { + data, ok := strings.CutPrefix(scanner.Text(), "data: ") + if !ok { + continue // SSE keep-alive/blank lines, comments, etc. + } + var msg transcript.Message + if err := json.Unmarshal([]byte(data), &msg); err != nil { + continue + } + printMessage(msg) + } + // The connection ending here — whether cleanly or as a network-level + // error — almost always just means the transcriptor-ai server stopped. + // We can't reliably tell that apart from a real connection problem, so + // don't be alarmist about it: this isn't a failure of transcript-tail + // itself, the way a startup connection error (above) is. + fmt.Fprintln(os.Stderr, "transcript-tail: stream ended (server stopped?)") + return nil +} + +func printMessage(msg transcript.Message) { + marker := " " + if !msg.IsFinal { + marker = "…" // still a partial hypothesis, may be revised + } + fmt.Printf("[%s%s] %s\n", msg.Track, marker, msg.Text) +} diff --git a/cmd/transcriptor-ai/main.go b/cmd/transcriptor-ai/main.go new file mode 100644 index 0000000..44d62aa --- /dev/null +++ b/cmd/transcriptor-ai/main.go @@ -0,0 +1,278 @@ +// Command transcriptor-ai orchestrates local, real-time meeting transcription: +// it drives audiotee for audio capture, qwen-asr for speech recognition, and +// serves the resulting transcript live over SSE. See CLAUDE.md for the full +// architecture and the reasoning behind it. +// +// This is a capture+VAD+ASR+SSE test harness: it runs audiotee, feeds +// detected speech segments to qwen_asr (one subprocess call per segment, one +// worker per track), and publishes the results both to stdout and to an SSE +// endpoint (`curl -N http:///events` is the terminal client for +// now — see CLAUDE.md on why SSE, not WebSocket). Post-processing (fuzzy +// glossary correction, LLM reread) isn't wired in yet. +package main + +import ( + "context" + "flag" + "fmt" + "net/http" + "os" + "os/signal" + "path/filepath" + "sync" + "syscall" + "time" + + "github.com/stephanetailland/transcriptor-ai/internal/asr" + "github.com/stephanetailland/transcriptor-ai/internal/capture" + "github.com/stephanetailland/transcriptor-ai/internal/transcript" + "github.com/stephanetailland/transcriptor-ai/internal/vad" +) + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "transcriptor-ai:", err) + os.Exit(1) + } +} + +func run() error { + audioteeBinary := flag.String("audiotee-binary", "", "path to the audiotee binary (optional; defaults to PATH then ~/bin/audiotee — see internal/capture)") + asrBinary := flag.String("asr-binary", "", "path to the qwen_asr binary (required)") + asrModelDir := flag.String("asr-model-dir", "", "path to the qwen_asr model directory (required)") + prompt := flag.String("prompt", "", "glossary biasing prompt passed to qwen_asr") + httpAddr := flag.String("http-addr", ":8420", "address to serve the live transcript SSE endpoint on") + transcriptFile := flag.String("transcript-file", "", "path to append the live transcript to as plain text (optional; not written if empty)") + captureSystem := flag.Bool("capture-system", true, "transcribe the system audio track (audiotee's system tap always runs regardless of this flag — disabling it only stops transcribing that track, see internal/capture.Config.CaptureMic doc)") + captureMic := flag.Bool("capture-mic", true, "capture and transcribe the microphone track") + flag.Parse() + + if *asrBinary == "" || *asrModelDir == "" { + return fmt.Errorf("both -asr-binary and -asr-model-dir are required") + } + if !*captureSystem && !*captureMic { + return fmt.Errorf("at least one of -capture-system or -capture-mic must be enabled") + } + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + systemPath := filepath.Join(os.TempDir(), "transcriptor-ai-system.pcm") + micPath := filepath.Join(os.TempDir(), "transcriptor-ai-mic.pcm") + + systemOut, err := os.Create(systemPath) + if err != nil { + return fmt.Errorf("create system output file: %w", err) + } + defer systemOut.Close() + + micOut, err := os.Create(micPath) + if err != nil { + return fmt.Errorf("create mic output file: %w", err) + } + defer micOut.Close() + + var fileWriter *transcript.FileWriter + if *transcriptFile != "" { + fileWriter, err = transcript.NewFileWriter(*transcriptFile) + if err != nil { + return err + } + defer fileWriter.Close() + fmt.Fprintln(os.Stderr, "transcript file ->", *transcriptFile) + } + + broadcaster := transcript.NewBroadcaster() + httpServer := &http.Server{Addr: *httpAddr, Handler: sseMux(broadcaster)} + httpErrs := make(chan error, 1) + go func() { + if err := httpServer.ListenAndServe(); err != nil && err != http.ErrServerClosed { + httpErrs <- err + } + }() + + const sampleRate = 16000 + c := capture.New(capture.Config{SampleRate: sampleRate, AudioteePath: *audioteeBinary, CaptureMic: *captureMic}) + if err := c.Start(ctx); err != nil { + return fmt.Errorf("start capture: %w", err) + } + + fmt.Fprintln(os.Stderr, "capturing... Ctrl+C to stop") + fmt.Fprintln(os.Stderr, "system track ->", systemPath) + fmt.Fprintln(os.Stderr, "mic track ->", micPath) + fmt.Fprintln(os.Stderr, "live transcript (SSE) -> curl -N http://"+httpAddrForDisplay(*httpAddr)+"/events") + + // Tee raw chunks to disk (for listening back later) while also feeding + // them to VAD. Buffered so the tee doesn't block segmentation. + tee := make(chan capture.Chunk, 64) + go func() { + defer close(tee) + for chunk := range c.Chunks() { + switch chunk.Track { + case capture.TrackSystem: + systemOut.Write(chunk.Data) + case capture.TrackMic: + micOut.Write(chunk.Data) + } + tee <- chunk + } + }() + + detector := vad.New(vad.DefaultConfig(sampleRate)) + segments := detector.Run(tee) + + // Two independent transcribers (2 tracks, 2 ASR engines — see + // CLAUDE.md), each processing its own track's segments sequentially. A + // fresh qwen_asr subprocess per segment is fast enough (~0.15-0.2x + // realtime measured) that this doesn't need to be more concurrent than + // that. + transcriber := asr.New(asr.Config{BinaryPath: *asrBinary, ModelDir: *asrModelDir, Prompt: *prompt}) + // transcribeTrack uses its own background context, not the + // shutdown-signal ctx above: segments still queued when Ctrl+C arrives + // (including the ones flushed by c.Stop() below) must still get + // transcribed during the graceful drain, not fail with "context + // canceled" because the shutdown signal already canceled ctx. + asrCtx := context.Background() + var systemSegs, micSegs chan vad.Segment + var wg sync.WaitGroup + if *captureSystem { + systemSegs = make(chan vad.Segment, 8) + wg.Add(1) + go func() { defer wg.Done(); transcribeTrack(asrCtx, transcriber, systemSegs, broadcaster, fileWriter) }() + } + if *captureMic { + micSegs = make(chan vad.Segment, 8) + wg.Add(1) + go func() { defer wg.Done(); transcribeTrack(asrCtx, transcriber, micSegs, broadcaster, fileWriter) }() + } + +loop: + for { + select { + case seg, ok := <-segments: + if !ok { + break loop + } + logSegment(seg) + dispatch(seg, systemSegs, micSegs) + case err := <-c.Errs(): + fmt.Fprintln(os.Stderr, "capture error:", err) + case err := <-httpErrs: + fmt.Fprintln(os.Stderr, "http server error:", err) + case <-ctx.Done(): + break loop + } + } + + // A graceful shutdown can take a few seconds (draining queued segments + // through qwen_asr, up to ~2s each on the 1.7B model) with no visible + // progress otherwise, which reads as "hung" — say so, and let a second + // Ctrl+C skip the wait and exit immediately instead of forcing the user + // to keep hitting it. + fmt.Fprintln(os.Stderr, "shutting down, waiting for in-flight transcriptions to finish (Ctrl+C again to force quit)...") + forceQuit := make(chan os.Signal, 1) + signal.Notify(forceQuit, syscall.SIGINT, syscall.SIGTERM) + go func() { + <-forceQuit + fmt.Fprintln(os.Stderr, "\nforcing immediate exit") + os.Exit(1) + }() + + c.Stop() + // Drain any segments flushed by Stop() closing the chunk stream. + for seg := range segments { + logSegment(seg) + dispatch(seg, systemSegs, micSegs) + } + + // Closing a nil channel panics — only close the ones actually created + // (i.e. for tracks that were enabled). + if systemSegs != nil { + close(systemSegs) + } + if micSegs != nil { + close(micSegs) + } + wg.Wait() // let in-flight/queued transcriptions finish before exiting, and + // before their results are published — otherwise a connected SSE client + // would miss the tail end of the transcript. + + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = httpServer.Shutdown(shutdownCtx) + + return nil +} + +// dispatch routes seg to the channel for its track. Sending to a nil +// channel (a disabled track — see -capture-system/-capture-mic) blocks +// forever in Go, so it must be checked, not just sent to unconditionally; +// a disabled track's segments are silently dropped (VAD still runs on +// every track's chunks regardless — negligible cost — but a disabled +// track never reaches ASR). +func dispatch(seg vad.Segment, systemSegs, micSegs chan<- vad.Segment) { + switch seg.Track { + case capture.TrackSystem: + if systemSegs != nil { + systemSegs <- seg + } + case capture.TrackMic: + if micSegs != nil { + micSegs <- seg + } + } +} + +func sseMux(b *transcript.Broadcaster) http.Handler { + mux := http.NewServeMux() + mux.Handle("/events", b) + return mux +} + +// httpAddrForDisplay turns a ":8420"-style listen address into something +// pasteable ("localhost:8420") for the printed curl hint. +func httpAddrForDisplay(addr string) string { + if len(addr) > 0 && addr[0] == ':' { + return "localhost" + addr + } + return addr +} + +func transcribeTrack(ctx context.Context, t *asr.Transcriber, segs <-chan vad.Segment, b *transcript.Broadcaster, fw *transcript.FileWriter) { + for seg := range segs { + text, err := t.Transcribe(ctx, seg.Data) + if err != nil { + fmt.Fprintf(os.Stderr, "[%s] asr error: %v\n", seg.Track, err) + continue + } + if text == "" { + continue + } + msg := transcript.Message{Track: seg.Track, Text: text, IsFinal: seg.IsFinal, Timestamp: seg.End} + b.Publish(msg) + if fw != nil { + // Loud, not fatal: losing one line to a transient write error + // shouldn't take down a live transcription session — but stay + // visible, since silent data loss would defeat the point of + // this file existing at all. + if err := fw.Write(msg); err != nil { + fmt.Fprintf(os.Stderr, "[%s] transcript file write error: %v\n", seg.Track, err) + } + } + + finality := "final" + if !seg.IsFinal { + finality = "partial" + } + fmt.Printf("[%s/%s] %s\n", seg.Track, finality, text) + } +} + +func logSegment(seg vad.Segment) { + kind := "final" + if !seg.IsFinal { + kind = "force-cut" + } + fmt.Fprintf(os.Stderr, "[%s] %-6s dur=%-6s bytes=%d (%s)\n", + seg.Start.Format("15:04:05.000"), seg.Track, seg.End.Sub(seg.Start).Round(10*time.Millisecond), len(seg.Data), kind) +} diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..4590c3a --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module github.com/stephanetailland/transcriptor-ai + +go 1.26.5 diff --git a/internal/asr/asr.go b/internal/asr/asr.go new file mode 100644 index 0000000..3f97d78 --- /dev/null +++ b/internal/asr/asr.go @@ -0,0 +1,77 @@ +// Package asr transcribes bounded PCM segments via qwen-asr +// (https://github.com/antirez/qwen-asr), invoked as a one-shot subprocess +// per segment rather than a long-lived daemon. +// +// This was verified empirically, not assumed: `qwen_asr --stdin --stream +// --silent` does not print incremental partial text as it processes a +// single stdin stream — it prints only the final transcription once done, +// even in --stream mode (that flag changes its internal chunked-encoding +// strategy, not what gets written to stdout). Measured latency including +// process startup and model load, on the 0.6B model, was ~0.15-0.2x +// realtime for a ~6s clip — comfortably fast enough to spawn fresh per VAD +// segment instead of keeping a persistent process fed over a pipe. +// Our own VAD segment boundaries (internal/vad) are therefore what stands +// in for "live" here: each vad.Segment.IsFinal maps directly to the +// SSE message's is_final field described in CLAUDE.md. +package asr + +import ( + "bytes" + "context" + "fmt" + "os" + "strconv" + "strings" +) + +// Config points at a built qwen_asr binary and a downloaded model +// directory. Neither has an established default location yet (unlike +// audiotee's ~/bin convention), so both are required. +type Config struct { + BinaryPath string + ModelDir string + // Prompt biases transcription toward glossary terms, e.g. + // "Preserve spelling: Kubernetes, PostgreSQL". Optional. + Prompt string + // Threads sets qwen_asr's -t flag. Zero means "let it pick" (all CPUs). + Threads int +} + +type Transcriber struct { + cfg Config + run runFunc // seam for testing without a real binary +} + +func New(cfg Config) *Transcriber { + return &Transcriber{cfg: cfg, run: runCommand} +} + +// Transcribe runs qwen_asr once against pcm (raw s16le, 16kHz, mono — the +// format both audiotee and internal/vad produce) and returns the +// transcribed text, trimmed. +func (t *Transcriber) Transcribe(ctx context.Context, pcm []byte) (string, error) { + args := []string{"-d", t.cfg.ModelDir, "--stdin", "--stream", "--silent"} + if t.cfg.Prompt != "" { + args = append(args, "--prompt", t.cfg.Prompt) + } + if t.cfg.Threads > 0 { + args = append(args, "-t", strconv.Itoa(t.cfg.Threads)) + } + + stdout, stderr, err := t.run(ctx, t.cfg.BinaryPath, args, pcm) + if err != nil { + return "", fmt.Errorf("qwen_asr: %w (stderr: %s)", err, bytes.TrimSpace(stderr)) + } + text := strings.TrimSpace(string(stdout)) + + if looksLikePromptLeak(text, t.cfg.Prompt) { + // Discard rather than return it as a genuine transcription — see + // leak.go. Logged here (not just silently dropped) since this + // package has no structured logger of its own; matches the plain + // stderr diagnostics the CLI already prints elsewhere. + fmt.Fprintf(os.Stderr, "asr: discarding segment, looks like a --prompt leak, not a transcription: %q\n", text) + return "", nil + } + + return text, nil +} diff --git a/internal/asr/leak.go b/internal/asr/leak.go new file mode 100644 index 0000000..0b07631 --- /dev/null +++ b/internal/asr/leak.go @@ -0,0 +1,61 @@ +package asr + +import ( + "strings" + "unicode" +) + +// looksLikePromptLeak reports whether text appears to be the model echoing +// the --prompt biasing content back verbatim instead of genuinely +// transcribing the segment's audio — a known failure mode of +// prompt-conditioned generation, confirmed via real testing against +// qwen_asr (see transcriptor-ui/CLAUDE.md, "Glossary prompt-leakage bug"): +// a whole segment came back as literally the glossary terms, twice, on two +// separate 5-minute test runs, even after switching to the tool's +// recommended "Preserve spelling: ..." prompt framing (which reduces but +// does not eliminate the risk). +// +// Heuristic: if most of text's words also appear among prompt's words, and +// there are enough of them to rule out a legitimate short mention of one +// or two glossary terms in real speech, treat it as a leak. Both observed +// real leaks were near-total reproductions of the prompt's term list, so a +// strict threshold (80% word overlap, at least 4 words) catches them +// without plausibly false-flagging a normal sentence that happens to +// mention "Claude" or "Mistral" once — normal speech has enough other +// words to keep the overlap ratio well below that. +func looksLikePromptLeak(text, prompt string) bool { + if prompt == "" || text == "" { + return false + } + textWords := normalizeWords(text) + if len(textWords) < 4 { + return false + } + promptWords := wordSet(normalizeWords(prompt)) + + matched := 0 + for _, w := range textWords { + if promptWords[w] { + matched++ + } + } + return float64(matched)/float64(len(textWords)) >= 0.8 +} + +// normalizeWords lowercases and splits on anything that isn't a letter or +// digit — unicode.IsLetter (not an ASCII-only check) so accented +// characters (French: é, è, ï, ç, ...) stay part of their word instead of +// being split into fragments. +func normalizeWords(s string) []string { + return strings.FieldsFunc(strings.ToLower(s), func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsDigit(r) + }) +} + +func wordSet(words []string) map[string]bool { + set := make(map[string]bool, len(words)) + for _, w := range words { + set[w] = true + } + return set +} diff --git a/internal/asr/runner.go b/internal/asr/runner.go new file mode 100644 index 0000000..32f8c52 --- /dev/null +++ b/internal/asr/runner.go @@ -0,0 +1,28 @@ +package asr + +import ( + "bytes" + "context" + "os/exec" + "syscall" +) + +// runFunc executes binary with args, feeding stdin, and returns +// stdout/stderr. It's a seam so tests can substitute a fake process instead +// of requiring the real qwen_asr binary and a multi-GB model on disk. +type runFunc func(ctx context.Context, binary string, args []string, stdin []byte) (stdout, stderr []byte, err error) + +func runCommand(ctx context.Context, binary string, args []string, stdin []byte) ([]byte, []byte, error) { + cmd := exec.CommandContext(ctx, binary, args...) + cmd.Stdin = bytes.NewReader(stdin) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + // Run qwen_asr in its own process group so the terminal's Ctrl+C + // (SIGINT to the whole foreground process group) doesn't kill an + // in-flight transcription directly. It's still cancelable through ctx, + // same as before — this only stops the shell from also signaling it. + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + err := cmd.Run() + return stdout.Bytes(), stderr.Bytes(), err +} diff --git a/internal/capture/capture.go b/internal/capture/capture.go new file mode 100644 index 0000000..6f010f8 --- /dev/null +++ b/internal/capture/capture.go @@ -0,0 +1,238 @@ +// Package capture wraps the audiotee subprocess and turns its two audio +// outputs (system audio on stdout, microphone on a FIFO) into a single +// stream of tagged chunks that downstream VAD/ASR stages can consume. +package capture + +import ( + "context" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "sync" + "syscall" + "time" +) + +// Track identifies which audio source a Chunk came from. +type Track string + +const ( + TrackSystem Track = "system" + TrackMic Track = "mic" +) + +// Chunk is a slice of raw PCM audio (16-bit signed, little-endian, mono, at +// Config.SampleRate) read off one of audiotee's two outputs. Timestamp is +// when this process read the chunk, not when audiotee captured it — audiotee +// does not emit per-chunk timestamps (see transcriptor-ai/CLAUDE.md), so this +// is an approximation good enough for live display, not sample-accurate +// realignment. +type Chunk struct { + Track Track + Data []byte + Timestamp time.Time +} + +// Config controls how the audiotee subprocess is launched. +type Config struct { + // AudioteePath is the path to the audiotee binary. If empty, it's + // resolved via PATH and then $HOME/bin/audiotee. + AudioteePath string + // SampleRate is passed to audiotee's --sample-rate flag. + SampleRate int + // ChunkBytes is the read buffer size used per Chunk. Must be even + // (16-bit samples). Defaults to 3200 bytes (~100ms at 16kHz mono). + ChunkBytes int + // CaptureMic controls whether audiotee is asked to also capture the + // mic track (--capture-mic --mic-output). audiotee has no equivalent + // flag to skip the system tap — it's always captured — so there is no + // "mic only" at this layer; a caller that only wants mic output must + // still receive (and can simply ignore) TrackSystem chunks. See + // CLAUDE.md for why this is a known, accepted limitation rather than + // something fixed here. + CaptureMic bool +} + +// Capturer runs audiotee and streams both its tracks as Chunks. +type Capturer struct { + cfg Config + cmd *exec.Cmd + fifoPath string + micFile *os.File + + chunks chan Chunk + errs chan error + + wg sync.WaitGroup + done chan struct{} // closed once both track readers have hit EOF +} + +// New creates a Capturer. Call Start to launch audiotee. +func New(cfg Config) *Capturer { + if cfg.ChunkBytes <= 0 { + cfg.ChunkBytes = 3200 + } + if cfg.ChunkBytes%2 != 0 { + cfg.ChunkBytes++ + } + return &Capturer{ + cfg: cfg, + chunks: make(chan Chunk, 64), + errs: make(chan error, 4), + } +} + +// Chunks returns the channel both tracks' audio is delivered on. Closed once +// both tracks have stopped delivering data (audiotee exited or Stop was +// called). +func (c *Capturer) Chunks() <-chan Chunk { return c.chunks } + +// Errs returns non-fatal errors encountered while reading either track. +func (c *Capturer) Errs() <-chan error { return c.errs } + +// Start resolves the audiotee binary, launches it with system+mic capture +// enabled, and begins streaming both tracks. It returns once audiotee has +// been launched; reading happens in background goroutines. +func (c *Capturer) Start(ctx context.Context) error { + audioteePath, err := resolveAudioteePath(c.cfg.AudioteePath) + if err != nil { + return err + } + + args := []string{"--sample-rate", fmt.Sprintf("%d", c.cfg.SampleRate)} + + var fifoPath string + if c.cfg.CaptureMic { + fifoPath = filepath.Join(os.TempDir(), fmt.Sprintf("transcriptor-ai-mic-%d.fifo", os.Getpid())) + if err := syscall.Mkfifo(fifoPath, 0o600); err != nil { + return fmt.Errorf("create mic fifo: %w", err) + } + c.fifoPath = fifoPath + args = append(args, "--capture-mic", "--mic-output", fifoPath) + } + + cmd := exec.CommandContext(ctx, audioteePath, args...) + cmd.Stderr = os.Stderr // audiotee's JSON logs, useful as-is while iterating + + stdout, err := cmd.StdoutPipe() + if err != nil { + c.cleanupFifo() + return fmt.Errorf("attach stdout pipe: %w", err) + } + + // Open the FIFO for reading in the background before starting audiotee: + // opening a FIFO blocks until the other end is opened too, so this must + // not block Start() itself, and audiotee (the writer) hasn't been + // spawned yet at this point. + var micOpened chan struct{} + var micFile *os.File + var micOpenErr error + if c.cfg.CaptureMic { + micOpened = make(chan struct{}) + go func() { + defer close(micOpened) + micFile, micOpenErr = os.OpenFile(fifoPath, os.O_RDONLY, 0) + }() + } + + if err := cmd.Start(); err != nil { + c.cleanupFifo() + return fmt.Errorf("start audiotee: %w", err) + } + c.cmd = cmd + + if c.cfg.CaptureMic { + <-micOpened + if micOpenErr != nil { + return fmt.Errorf("open mic fifo: %w", micOpenErr) + } + c.micFile = micFile + } + + c.done = make(chan struct{}) + c.wg.Add(1) + go c.readTrack(TrackSystem, stdout) + + if c.cfg.CaptureMic { + c.wg.Add(1) + go c.readTrack(TrackMic, micFile) + } + + go func() { + c.wg.Wait() + close(c.chunks) + close(c.done) + }() + + return nil +} + +func (c *Capturer) readTrack(track Track, r io.Reader) { + defer c.wg.Done() + buf := make([]byte, c.cfg.ChunkBytes) + for { + n, err := io.ReadFull(r, buf) + if n > 0 { + data := make([]byte, n) + copy(data, buf[:n]) + c.chunks <- Chunk{Track: track, Data: data, Timestamp: time.Now()} + } + if err != nil { + if err != io.EOF && err != io.ErrUnexpectedEOF { + select { + case c.errs <- fmt.Errorf("%s track: %w", track, err): + default: + } + } + return + } + } +} + +// Stop terminates audiotee gracefully (SIGINT, matching the fix in audiotee +// itself so shutdown doesn't hang) and cleans up the FIFO. +func (c *Capturer) Stop() error { + if c.cmd != nil && c.cmd.Process != nil { + _ = c.cmd.Process.Signal(syscall.SIGINT) + } + if c.done != nil { + // audiotee exiting closes its stdout pipe and the mic FIFO write + // end, which is what makes readTrack see EOF — wait for that before + // reaping the process. Calling cmd.Wait() first can close the pipe + // out from under an in-progress read (see os/exec's StdoutPipe doc). + <-c.done + } + if c.cmd != nil && c.cmd.Process != nil { + _ = c.cmd.Wait() + } + if c.micFile != nil { + _ = c.micFile.Close() + } + c.cleanupFifo() + return nil +} + +func (c *Capturer) cleanupFifo() { + if c.fifoPath != "" { + _ = os.Remove(c.fifoPath) + } +} + +func resolveAudioteePath(configured string) (string, error) { + if configured != "" { + return configured, nil + } + if p, err := exec.LookPath("audiotee"); err == nil { + return p, nil + } + home, err := os.UserHomeDir() + if err == nil { + candidate := filepath.Join(home, "bin", "audiotee") + if info, statErr := os.Stat(candidate); statErr == nil && !info.IsDir() { + return candidate, nil + } + } + return "", fmt.Errorf("audiotee binary not found on PATH or in ~/bin — build it via audiotee/scripts/build-signed.sh") +} diff --git a/internal/transcript/transcript.go b/internal/transcript/transcript.go new file mode 100644 index 0000000..2fb89cf --- /dev/null +++ b/internal/transcript/transcript.go @@ -0,0 +1,112 @@ +// Package transcript defines the live transcript message format (see +// CLAUDE.md) and a Broadcaster that fans a single stream of messages out to +// any number of SSE clients — the merge stage's output contract. +// +// Ordering note: messages are broadcast in the order transcribeTrack +// produces them, not re-sorted by Timestamp. With two tracks processed +// independently, a slower track could in principle publish slightly out of +// chronological order relative to the other. Measured ASR latency +// (~0.15-0.2x realtime) is small relative to segment spacing (3-5s), so +// this hasn't been observed in practice — add a small reordering buffer +// here if it ever becomes visible, not before. +package transcript + +import ( + "encoding/json" + "fmt" + "net/http" + "sync" + "time" + + "github.com/stephanetailland/transcriptor-ai/internal/capture" +) + +// Message is one transcript segment, matching the SSE JSON format +// documented in CLAUDE.md. +type Message struct { + Track capture.Track `json:"track"` + Text string `json:"text"` + IsFinal bool `json:"is_final"` + Timestamp time.Time `json:"timestamp"` +} + +// clientBuffer bounds how many undelivered messages a slow SSE client can +// accumulate before new messages are dropped for it, so one stalled client +// can't back-pressure the whole pipeline. +const clientBuffer = 32 + +// Broadcaster fans Published messages out to every currently-connected SSE +// client and doubles as the http.Handler for the SSE endpoint. +type Broadcaster struct { + mu sync.Mutex + clients map[chan Message]struct{} +} + +func NewBroadcaster() *Broadcaster { + return &Broadcaster{clients: make(map[chan Message]struct{})} +} + +// Publish sends msg to every connected client. Non-blocking: a client whose +// buffer is full misses the message rather than stalling the sender. +func (b *Broadcaster) Publish(msg Message) { + b.mu.Lock() + defer b.mu.Unlock() + for ch := range b.clients { + select { + case ch <- msg: + default: + } + } +} + +func (b *Broadcaster) subscribe() chan Message { + ch := make(chan Message, clientBuffer) + b.mu.Lock() + b.clients[ch] = struct{}{} + b.mu.Unlock() + return ch +} + +func (b *Broadcaster) unsubscribe(ch chan Message) { + b.mu.Lock() + delete(b.clients, ch) + b.mu.Unlock() +} + +// ServeHTTP streams messages to the client as Server-Sent Events until the +// client disconnects. See CLAUDE.md for why SSE over WebSocket: the flow is +// one-directional, and this needs no client library beyond EventSource in a +// browser or a plain HTTP client in a terminal (e.g. `curl -N`). +func (b *Broadcaster) ServeHTTP(w http.ResponseWriter, r *http.Request) { + flusher, ok := w.(http.Flusher) + if !ok { + http.Error(w, "streaming unsupported", http.StatusInternalServerError) + return + } + + w.Header().Set("Content-Type", "text/event-stream") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("Connection", "keep-alive") + w.WriteHeader(http.StatusOK) + flusher.Flush() + + ch := b.subscribe() + defer b.unsubscribe(ch) + + for { + select { + case msg, ok := <-ch: + if !ok { + return + } + data, err := json.Marshal(msg) + if err != nil { + continue + } + fmt.Fprintf(w, "data: %s\n\n", data) + flusher.Flush() + case <-r.Context().Done(): + return + } + } +} diff --git a/internal/transcript/writer.go b/internal/transcript/writer.go new file mode 100644 index 0000000..03b67df --- /dev/null +++ b/internal/transcript/writer.go @@ -0,0 +1,49 @@ +package transcript + +import ( + "fmt" + "io" + "os" + "sync" +) + +// FileWriter appends each Message to a text file as a human-readable line, +// flushed to disk immediately. Unlike Broadcaster.Publish (which drops +// messages for a slow/stalled SSE client on purpose), FileWriter never +// drops — this is the durable record a crash shouldn't lose data from, so +// every Write is synchronous and fsync'd before returning. +type FileWriter struct { + mu sync.Mutex + f *os.File +} + +// NewFileWriter opens path for appending, creating it if needed. Appending +// (not truncating) means restarting transcriptor-ai against the same path +// continues the same transcript file rather than discarding it. +func NewFileWriter(path string) (*FileWriter, error) { + f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) + if err != nil { + return nil, fmt.Errorf("open transcript file: %w", err) + } + return &FileWriter{f: f}, nil +} + +// Write appends one line for msg and fsyncs before returning. +func (fw *FileWriter) Write(msg Message) error { + fw.mu.Lock() + defer fw.mu.Unlock() + line := fmt.Sprintf("[%s] [%s] %s\n", msg.Timestamp.Format("15:04:05"), msg.Track, msg.Text) + if _, err := io.WriteString(fw.f, line); err != nil { + return fmt.Errorf("write transcript line: %w", err) + } + if err := fw.f.Sync(); err != nil { + return fmt.Errorf("sync transcript file: %w", err) + } + return nil +} + +func (fw *FileWriter) Close() error { + fw.mu.Lock() + defer fw.mu.Unlock() + return fw.f.Close() +} diff --git a/internal/vad/vad.go b/internal/vad/vad.go new file mode 100644 index 0000000..86ba2c3 --- /dev/null +++ b/internal/vad/vad.go @@ -0,0 +1,217 @@ +// Package vad turns a stream of raw audio chunks into bounded speech +// segments using a simple energy-threshold heuristic — no ML model. See +// transcriptor-ai/CLAUDE.md for why: no Go/Swift ML ecosystem was mature +// enough to justify the integration cost for this. Revisit only if +// segmentation quality proves to be the actual bottleneck. +package vad + +import ( + "encoding/binary" + "math" + "time" + + "github.com/stephanetailland/transcriptor-ai/internal/capture" +) + +// Segment is a bounded span of speech audio ready for ASR. +type Segment struct { + Track capture.Track + Data []byte // raw PCM, concatenation of the contributing chunks + Start time.Time + End time.Time + // IsFinal is true when silence closed the utterance, false when the + // segment was force-cut at MaxSegmentDuration and more audio for the + // same utterance follows in the next segment (which starts with + // Config.Overlap worth of audio repeated for ASR context continuity). + IsFinal bool +} + +// Config controls the energy-threshold heuristic. All durations are +// expressed in wall-clock time; SampleRate converts them to byte counts. +type Config struct { + SampleRate int + + // SilenceThresholdDB is the RMS level (dBFS, 0 = full scale) below which + // a chunk is considered silence. Real speech in testing sits around -40 + // to -20 dBFS; captured silence sits around -90 dBFS. Needs empirical + // tuning against real meeting audio — this default is a starting point, + // not a validated value. + SilenceThresholdDB float64 + + // SilenceHangover is how long silence must persist after speech before + // the segment is closed as final (IsFinal: true). + SilenceHangover time.Duration + + // MinSpeechDuration is the minimum contiguous voiced audio required + // before a segment starts accumulating, to filter transient noise + // (clicks, pops) rather than opening a segment on every blip. + MinSpeechDuration time.Duration + + // MaxSegmentDuration force-cuts a segment even mid-speech, so a long + // monologue still produces incremental, timely output instead of one + // unbounded segment. + MaxSegmentDuration time.Duration + + // Overlap is how much trailing audio is carried into the next segment + // when force-cut, for ASR context continuity across the cut. Tradeoff: + // the carried-over audio gets transcribed again, and both copies are + // published — a nonzero Overlap produces visible duplicate text at + // every force-cut boundary (confirmed in real testing: this is worse + // than the word occasionally getting cut in half at a hard boundary, + // which is why DefaultConfig uses 0). Only raise this if paired with + // dedup logic on the published text — never on its own. + Overlap time.Duration +} + +// DefaultConfig returns starting-point thresholds; tune against real audio. +func DefaultConfig(sampleRate int) Config { + return Config{ + SampleRate: sampleRate, + SilenceThresholdDB: -40, + SilenceHangover: 600 * time.Millisecond, + MinSpeechDuration: 200 * time.Millisecond, + MaxSegmentDuration: 4 * time.Second, + Overlap: 0, + } +} + +func (c Config) bytesForDuration(d time.Duration) int { + // 16-bit mono samples: 2 bytes/sample. + samples := int(float64(c.SampleRate) * d.Seconds()) + n := samples * 2 + if n%2 != 0 { + n++ + } + return n +} + +// Detector runs the heuristic independently per track (system and mic have +// separate state) over a shared, tagged chunk stream. +type Detector struct { + cfg Config + states map[capture.Track]*trackState +} + +func New(cfg Config) *Detector { + return &Detector{cfg: cfg, states: make(map[capture.Track]*trackState)} +} + +type trackState struct { + inSpeech bool + buffer []byte + segStart time.Time + + // candidate holds voiced audio not yet promoted to a real segment, + // while we wait to confirm MinSpeechDuration. + candidate []byte + candidateStart time.Time + + silenceSince time.Time // zero value means "not currently in a silence run" + lastChunkAt time.Time // timestamp of the most recently processed chunk +} + +// Run consumes chunks until the channel closes, emitting Segments on the +// returned channel. Any in-progress segment is flushed (as final, since +// capture ended) before the output channel closes. +func (d *Detector) Run(chunks <-chan capture.Chunk) <-chan Segment { + out := make(chan Segment, 16) + go func() { + defer close(out) + for chunk := range chunks { + d.process(chunk, out) + } + for track, st := range d.states { + if st.inSpeech && len(st.buffer) > 0 { + out <- Segment{Track: track, Data: st.buffer, Start: st.segStart, End: st.lastChunkAt, IsFinal: true} + } + } + }() + return out +} + +func (d *Detector) process(chunk capture.Chunk, out chan<- Segment) { + st, ok := d.states[chunk.Track] + if !ok { + st = &trackState{} + d.states[chunk.Track] = st + } + + speech := dbFS(chunk.Data) >= d.cfg.SilenceThresholdDB + st.lastChunkAt = chunk.Timestamp + + switch { + case speech && !st.inSpeech: + // Possible start of an utterance — buffer as a candidate until + // MinSpeechDuration is confirmed, to filter transient noise. + if len(st.candidate) == 0 { + st.candidateStart = chunk.Timestamp + } + st.candidate = append(st.candidate, chunk.Data...) + if chunk.Timestamp.Sub(st.candidateStart) >= d.cfg.MinSpeechDuration { + st.inSpeech = true + st.segStart = st.candidateStart + st.buffer = st.candidate + st.candidate = nil + st.silenceSince = time.Time{} + } + + case speech && st.inSpeech: + st.buffer = append(st.buffer, chunk.Data...) + st.silenceSince = time.Time{} // speech resumed, cancel any pending hangover + if chunk.Timestamp.Sub(st.segStart) >= d.cfg.MaxSegmentDuration { + out <- Segment{Track: chunk.Track, Data: st.buffer, Start: st.segStart, End: chunk.Timestamp, IsFinal: false} + st.buffer = tail(st.buffer, d.cfg.bytesForDuration(d.cfg.Overlap)) + st.segStart = chunk.Timestamp.Add(-d.cfg.Overlap) + } + + case !speech && st.inSpeech: + if st.silenceSince.IsZero() { + st.silenceSince = chunk.Timestamp + } + st.buffer = append(st.buffer, chunk.Data...) // keep brief trailing silence for natural cutoff + if chunk.Timestamp.Sub(st.silenceSince) >= d.cfg.SilenceHangover { + out <- Segment{Track: chunk.Track, Data: st.buffer, Start: st.segStart, End: chunk.Timestamp, IsFinal: true} + st.inSpeech = false + st.buffer = nil + st.silenceSince = time.Time{} + } + + default: // !speech && !st.inSpeech + st.candidate = nil + } +} + +// dbFS returns the RMS level of a 16-bit little-endian PCM buffer in dBFS +// (0 = full scale for int16). Silence (all-zero input) returns -infinity, +// which compares correctly against any real threshold. +func dbFS(data []byte) float64 { + n := len(data) / 2 + if n == 0 { + return math.Inf(-1) + } + var sumSquares float64 + for i := 0; i+1 < len(data); i += 2 { + s := int16(binary.LittleEndian.Uint16(data[i : i+2])) + v := float64(s) + sumSquares += v * v + } + rms := math.Sqrt(sumSquares / float64(n)) + if rms == 0 { + return math.Inf(-1) + } + return 20 * math.Log10(rms/32768) +} + +func tail(data []byte, n int) []byte { + if n <= 0 { + return nil + } + if n >= len(data) { + out := make([]byte, len(data)) + copy(out, data) + return out + } + out := make([]byte, n) + copy(out, data[len(data)-n:]) + return out +} diff --git a/scripts/build-dist.sh b/scripts/build-dist.sh new file mode 100755 index 0000000..a78d86b --- /dev/null +++ b/scripts/build-dist.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Builds transcriptor-ai and its qwen_asr dependency into dist/, ready to be +# embedded by transcriptor-ui's Xcode build phase. Models are NOT part of +# this — those are fetched at runtime by the app (see CLAUDE.md), only +# binaries are built/staged here. +# +# Usage: scripts/build-dist.sh + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$REPO_ROOT" + +DIST_DIR="$REPO_ROOT/dist" +QWEN_ASR_DIR="$REPO_ROOT/third_party/qwen-asr" + +echo "Ensuring qwen-asr submodule is present (pinned commit)..." +git submodule update --init --recursive third_party/qwen-asr + +echo "Building qwen_asr..." +make -C "$QWEN_ASR_DIR" blas + +echo "Building transcriptor-ai..." +mkdir -p "$DIST_DIR" +go build -o "$DIST_DIR/transcriptor-ai" ./cmd/transcriptor-ai + +cp "$QWEN_ASR_DIR/qwen_asr" "$DIST_DIR/qwen_asr" + +echo "" +echo "dist/ ready:" +ls -la "$DIST_DIR" diff --git a/third_party/qwen-asr b/third_party/qwen-asr new file mode 160000 index 0000000..b00b789 --- /dev/null +++ b/third_party/qwen-asr @@ -0,0 +1 @@ +Subproject commit b00b789b17051aea61e9717458171100662318a4