AI meeting notes, recaps and operations companion

Problem

A technical team kept losing the decisions and commitments made in meetings, because notes lived in recordings and long transcripts nobody reread. People went into meetings without context. Recording relied on a paid cloud service, so audio and transcripts left the company, and its summaries did not match how the team tracked actions. Separately, staff had no quick way to check whether a link or an email was malicious.

Goal

Capture every meeting automatically; deliver a structured recap to the right place (the team, a private message or the project tracker) even when the transcript arrives late; let each person approve their own action items; brief people before meetings; answer questions from past meetings; and do it with audio kept on company hardware where that is required.

My role

Sole designer and builder of all three systems: the scheduled recap pipeline, the chat-native operations companion (including its threat-intelligence engine), and the self-hosted notetaker with its native audio helper.

Solution

Three builds that cover the whole meeting lifecycle.

A scheduled recap pipeline: a serverless polling job detects ended meetings, gathers the best content available, has an LLM write a structured recap and routes it by meeting type, with deduplication and deferred retries.

A chat-native operations companion: recaps with per-owner approval of action items, briefs before meetings, a question-answering companion over past meetings, uptime alerts, and a multi-source link and email threat scanner with an AI-written verdict.

A self-hosted notetaker: a browser bot joins scheduled calls, captures audio without echo, transcribes on the device, writes a structured summary and stores it encrypted, with LLM usage as the only recurring cost.

How it works

Capture (self-hosted notetaker).

A background service watches a local meeting schedule and, shortly before the start, launches a browser bot that joins as a named participant and shows a consent notice.

Inside the meeting page, the bot taps the remote participants' media streams with the Web Audio API and streams raw audio to the host, so nothing plays through speakers and nothing echoes. A native macOS helper can instead capture specific applications' audio through process taps.

Audio is resampled and transcribed on the device with a local Whisper model.

Records are encrypted with AES-256-GCM using a key held in the operating system's keychain, pruned on a retention schedule and viewable in a local dashboard. Bounded concurrency, attempt caps, success-only deduplication and a sleep inhibitor keep unattended runs reliable.

Recap and routing.

A scheduled function regularly lists meetings that ended within a recent time window and claims each one in a document store with a "processing" status. Finished or in-progress meetings are skipped, and only failed ones are retried.

It looks up the calendar event for the real title and any attached notes, maps participant names, and fetches content through a chain of fallback sources: the transcript, then an attached notes document, then a time-window search for notes.

An LLM turns the content into a fixed structure (summary, per-person updates, decisions and action items) through forced structured output. Long meetings are summarised in chunks and combined, with prompt caching to cut cost.

A routing table keyed on the meeting title sends the recap to a threaded post in a team space, a private direct message or a project-tool status update, so private meetings stay private.

If no content exists yet, a placeholder is posted and the meeting is saved as "awaiting content". Later runs edit the placeholder in place when content appears, and give up with a clear message after a set number of attempts.

Operations companion.

Extracted action items go to each owner privately with approve and reject controls, so nothing is created on someone's behalf without their say.

Before a meeting, a brief is assembled from the calendar and earlier recaps. A chat companion answers questions from stored recaps and the calendar, with access checks on who may see what.

An uptime monitor checks key services and alerts in chat.

A dependency-injected threat engine checks a submitted link or email against several reputation services in parallel, adds domain-age, DNS blocklist and TLS checks, analyses email authentication results, and has an LLM write a plain-language verdict. Each service sits behind its own circuit breaker, results are cached, calls are rate-limited, and fetched URLs are protected against server-side request forgery.

Technical challenges

Clean audio from a browser meeting with no echo and without muting anyone, and detecting when a waiting bot has been admitted.

Transcripts arrive minutes to hours late: deferred processing with a visible placeholder edited in place instead of duplicate posts.

No double processing across overlapping scheduled runs (claim records).

Readable titles and attached notes that the meeting API does not carry (a calendar lookup).

Stopping invented or unwanted action items from becoming work (per-owner approval).

A consistent summary shape, and transcripts longer than a model's context (map-reduce).

Keeping a threat scanner useful when individual intelligence services are slow or down, and fetching user-submitted URLs safely.

Encryption and key management at rest; unattended reliability through sleep, concurrency limits and retries; and end-to-end testing with synthetic audio instead of real meetings.

Capabilities

Automatic joining of scheduled video meetings
Echo-free in-browser capture with a per-application fallback
On-device transcription
Structured recaps with per-person commitments, decisions and action items
Fallback content sources and title-based routing to team, private or project destinations
Placeholder-then-edit delivery, deduplication and bounded retries
Per-owner action approval
Briefs before meetings and questions answered from past meetings
Uptime alerts
Link and email threat scanning with an AI verdict
An encrypted, retention-managed archive with a local dashboard

Technology

Impact

Every meeting produced a consistent recap in the right place without anyone writing notes. Commitments became work only with their owner's sign-off, people arrived briefed, and staff got fast, readable security verdicts. The self-hosted notetaker was built to replace a paid, cloud-hosted recorder with an in-house tool whose only recurring cost is LLM usage, keeping audio and transcripts on company hardware. No metrics are claimed.

Skills demonstrated

Serverless and event-driven pipeline design
Integration with workspace APIs through delegated authentication
LLM structured output and long-context handling
Idempotent state machines and retry design for late data
Routing and privacy design
Human-in-the-loop workflows
Browser automation and real-time audio engineering
On-device ML inference
Applied cryptography
Security engineering (threat intelligence, email authentication, SSRF defence)
Resilience patterns (circuit breakers, caching, rate limiting)