Local multilingual video and live-stream summarizer

Problem

Catching up on long recorded talks, meetings and broadcasts in several regional languages takes hours of listening, and cloud transcription costs money and sends audio elsewhere.

Goal

Hand over a video or audio recording you already hold and get back a short, honest English summary, produced mostly on the local machine for free, with uncertain names flagged and unclear stretches stated rather than papered over.

My role

Sole author: the pipeline design, the transcription quality checks, the optional cloud fallback and the portable packaging.

Solution

An AI assistant skill backed by a Python pipeline: take only the audio, convert it for speech recognition, transcribe it on the device with an open speech model tuned for Apple Silicon, detect where the transcript is unreliable, and hand a time-stamped transcript with explicit unclear markers to the assistant to summarise. A careful mode sends audio to a cloud model in chunks for better spelling of local names.

How it works

Audio only. The pipeline works from the audio track alone and converts it to the format the speech model expects.

Local transcription. Transcription runs on the device with conditioning on previous text turned off (which stops the model looping) and an initial prompt built from the recording's title and any names the user supplies, to steer the spelling of names.

Quality checks. Every segment is scored. Segments with no letters, or very few words stretched over many seconds, are treated as the model inventing phrases over music or silence. Long wordless gaps, repeating four-word loops and high compression ratios are reported as unclear stretches.

Summary input. The transcript is grouped into blocks of about half a minute, with markers where there was no speech, and handed to the assistant, which writes the English summary and marks uncertain names.

Careful mode. Fixed-length chunks go to a cloud multimodal model, and their timestamps are shifted back into place.

Precise results. The script prints one machine-readable result line with distinct exit codes (no speech found, not available yet, and so on), so the assistant can respond precisely.

Portable toolchain. All tools, the model and caches live inside one mountable disk image that opens for a run and closes afterwards.

Technical challenges

Speech models hallucinate confidently over music and silence, and their own confidence scores miss it, so custom checks were needed.

Code-switching between languages, and local proper nouns.

Keeping a heavy ML toolchain portable across machines on an external drive.

Capabilities

Audio-only processing of recorded video
On-device speech recognition
Hallucination and repetition detection
Unclear-stretch reporting
Optional chunked cloud transcription
English summaries of multilingual speech, with uncertain names flagged
Machine-readable results and exit codes
A self-installing portable toolchain

Technology

Impact

A long recording becomes a short summary in a small fraction of its running time, at no cost on the default path, with honest flags wherever the audio could not be understood.

Skills demonstrated

Applied speech ML
Reducing hallucination in speech recognition and LLMs
Media pipelines
Robust handling of child processes and their lifecycle
API integration
Local-first, privacy-aware design
Portable environment packaging