MemoScribe

Turn an iPhone voice memo into a plain-text transcript — locally, on-device.

Local-first · Python + uv · Apple M4 Pro (Metal) · Stage: MVP — audio → text

Milestones

✓ Project scaffolding uv project, Python 3.12, mlx-whisper + Metal GPU verified, ffmpeg installed.
✓ Simple file transcription One voice memo (.m4a) transcribed to text via Whisper, locally on the GPU.
✓ Saving transcript to a file Written to transcripts/<name>.txt behind a single swappable seam.
✓ Testing & eval (no-regression) One end-to-end pytest checking WER ≤ 10% vs a human-verified reference. Doubles as the eval seed.
→ Make it a real tool — CLI argumentnext Accept any file path instead of the hardcoded one; add a friendly-error negative test for bad input.
□ Speaker diarization — "who spoke what?" Open questions: how to label/disambiguate speakers, and how to store metadata about who the speakers are (see below).
□ RAG / style model · maybe iOS app Long-term: use transcripts for retrieval or to learn the user's conversational style.

What it does

You record a conversation in the iPhone Voice Memos app. MemoScribe takes that audio file and produces a text transcript — running entirely on your own machine, no cloud. The current goal is a single thing done correctly: one audio file in, one accurate .txt out.

The flow (MVP)

Everything from the blue box rightward runs on your Mac. Nothing leaves the device.

Key decisions

Local-firstno cloud APIs

Privacy of personal conversations, zero cost, and a better learning exercise. The M4 Pro (48 GB) is more than capable of running Whisper and, later, mid-size LLMs.

Whispertranscription engine

The de-facto open-source speech-to-text model. On Apple Silicon, an MLX-backed build uses the GPU (Metal) natively — fastest path on this hardware.

Python + uvlanguage & tooling

Best ecosystem for ML/audio. uv gives reproducible, fast dependency management. A potential iOS app is a separate, later concern.

One step at a timebuild philosophy

Each change does one specific, understood thing. Correctness of the basic conversion comes before any added capability.

Roadmap

Now Audio → text
correct transcription

Later Speaker diarization
who said what

Started Evaluation
WER regression test

Someday RAG / style model
+ maybe iOS

Dashed = not built yet. Detail gets added to this page only when each phase is actually started.

Under the hood MVP working

Model: OpenAI Whisper (small) — speech-to-text neural net.
Runtime: mlx-whisper — runs Whisper on Apple MLX / Metal (the GPU).
Where the model comes from: downloaded once from Hugging Face, repo mlx-community/whisper-small-mlx (MLX-converted weights).
Model cache: stored under ~/.cache/huggingface/ — re-used on every later run, no re-download.
Audio decode: ffmpeg turns the .m4a into raw 16 kHz samples Whisper expects. (Only system dependency — reversible; the one swap point if we go zero-system-dependency.)
Entry point: transcribe.py — currently one hardcoded file in, transcript saved out.
Output: transcript written to transcripts/<name>.txt. A single save_transcript() seam owns where files go — repoint it later (other folder, S3) without touching transcription.

Testing & eval 1 test passing

Runner: pytest (dev dependency). Run with uv run pytest.
Metric: Word Error Rate via jiwer, normalized to ignore punctuation & casing — only word correctness counts.
Why a threshold, not exact match: tolerates harmless wording drift when we swap models, yet still catches a broken pipeline or a much worse model. This same test is the seed of the eval suite.
Current result: the small model scores 0% WER on the clean fixture. (Easy audio — the real signal from WER comes on harder clips later.)

Open questions

Cleanest way to get memos from iPhone → Mac (AirDrop vs. iCloud-synced Voice Memos library).
How to disambiguate speakers (diarization) — likely pyannote.audio, deferred.
How to write evals for transcription quality — approach still to be decided.