How I Taught a Local LLM to See the Meeting

Today I downloaded a meeting recording and transcript from Fellow, handed both to a command-line tool, and got back the kind of notes I had been trying to produce for months: not just what people said, but what they were actually looking at when they said it.

The transcript knew the team was discussing an onboarding flow. The video showed which screen they meant, which option was selected, and what changed between one version and the next. Put together, those signals made the notes substantially more useful.

The satisfying part is that this now feels simple:

lc-vision analyze <fellow-video-url> \
  --title "Product Design Review" \
  --transcript meeting-transcript.txt

But it did not begin as a clean pipeline. It began with a spare Mac, a local model, and a long string of experiments that were fast, promising, confusing, occasionally wrong, and—eventually—useful.

This is the story of that process: how I chose what to test, how I learned to read failures, and how the resulting system now gives our Ledger agent better evidence for better notes.

The flow that works today

The current workflow has five distinct jobs:

Fellow gives us two complementary sources: the recording and the transcript. The Vision service, running on a Mac on our network, analyzes both. The vision-cli package gives people and agents a small interface for submitting a video, following the job, and asking focused follow-up questions.

The output is not a finished meeting note. It is an evidence bundle: chapters, decisions, action items, questions, relevant frames, on-screen content, timestamps, and confidence signals.

That distinction matters. Vision does not replace Ledger. It gives Ledger another sense.

Ledger already knows how to gather the meeting context, reconcile identities, preserve decisions, and publish a structured note in Notion. Vision now enters that pipeline as an enrichment stage. Ledger can use it to resolve vague transcript references—“this version,” “the card on the left,” “that number”—and then synthesize the combined evidence into the note.

The video is synthesis fuel, not a new section someone has to read.

Why the transcript was not enough

Transcripts are excellent at language and terrible at deixis: all the little words whose meaning depends on what is visible.

“Move this under that.”

“The second version feels better.”

“That total is wrong.”

A transcript preserves those sentences while discarding the thing that makes them meaningful. A meeting recording contains that missing context, but only if the system can find the few frames that matter without narrating every webcam tile and transition animation.

Here is a sanitized recreation of the difference:

The point is not to produce longer notes. It is to replace ambiguity with grounded detail.

That became the product test for every experiment: does this help a future reader understand what happened and act on it?

I started with a spare Mac

I wanted the recordings to stay on hardware we control, and I wanted experimentation to be cheap enough that I could run it repeatedly. Apple Silicon made a local pipeline plausible: decode the video, transcribe audio, select frames, run a multimodal model, and synthesize the result without sending the meeting to another hosted inference API.

The first architecture was deliberately small:

• ffmpeg for audio extraction and scene detection
• Whisper for transcription when a transcript was not already available
• a local vision-language model for image and text analysis
• a Bun service to coordinate the work
• a queue so heavy inference stayed observable and recoverable

I did not begin by designing the final system. I began by choosing one real meeting and defining what “better” meant: useful chapter boundaries, correct action items, no invented decisions, grounded visual observations, and a runtime that fit into an actual workday.

Then I changed one meaningful variable at a time and read the output.

The model search was not a leaderboard

It is tempting to turn local model selection into a spreadsheet: tokens per second, memory use, parameter count, context size. Those numbers matter, but they are not the job.

The job was trustworthy meeting understanding.

Phi was fast because it did less

One Phi run finished in roughly half the time of the baseline. That looked like a breakthrough until I read the output.

It had produced only a handful of coarse chapters. Most structured extraction was empty. Action-item recall collapsed. The pipeline was not more efficient; it was skipping the hard work.

That established an early rule: runtime is only meaningful after quality passes.

Qwen was genuinely promising

Qwen was faster, lighter, and sometimes produced excellent rewrites. It also exposed a more subtle problem: model capability and production reliability are different things.

Longer structured responses sometimes arrived as malformed JSON. The pipeline would silently fall back to raw transcript fragments, making a capable model look worse than it was. After changing the output contract to newline-delimited JSON and accounting for every input, the quality improved dramatically.

But other stages still revealed a gap. Qwen did well when the contract was mechanical and explicit. It struggled more when the task required unstated judgment: deciding whether two items represented the same underlying decision, filtering plausible-but-unimportant remarks, or recognizing that a demo contained explanation rather than commitments.

The lesson was not “Qwen is bad.” It was: evaluate the exact kind of judgment your pipeline needs.

A parser bug cost me hours

At one point a model returned a valid range-shaped structure such as [[34, 34], [37, 37]], while my parser expected flat indexes like [34, 37]. The request succeeded. The JSON was valid. My code filtered the result down to nothing.

I spent hours forming theories about the model being overly conservative before reading the raw response closely enough to see the mismatch.

That became the most reusable debugging rule in the project:

When an LLM stage unexpectedly returns zero, read the raw response before changing the model or the prompt.

Parser failures can masquerade as model failures. A metric cannot tell you which one you are looking at.

Three model processes made everything worse

I also tried the obvious throughput optimization: run three MLX model processes and distribute work across them.

The full pipeline became only about five percent faster—and captured roughly forty-five percent fewer useful items.

On a shared Apple GPU, duplicating weights and interleaving work across separate processes created memory pressure and scheduling overhead without giving me three independent GPUs. The correct shape was one weight set, a legible queue, and a serving layer capable of batching concurrent requests.

This was a useful reminder that “parallel” is not automatically “faster,” especially when the scarce resource is shared.

The serving layer mattered as much as the model

I initially dismissed one serving option after a shallow research pass, then revisited it when the architecture kept pointing in that direction. A smoke test looked great. The real workload exposed thinking-mode leakage, slow long prompts, and unstable structured output.

Later, another configuration failure turned out to be mine: an enable_thinking option was nested in the wrong place. Once again, the component I blamed was not the component that failed.

The production system eventually moved to OMLX because it behaved cleanly on the dimensions that mattered here: multimodal requests, structured output, concurrency isolation, and one local OpenAI-compatible interface for both image and text work.

Then I deliberately moved from the smaller Gemma variant to a larger 12B 8-bit Gemma 4 model. It was slower. The notes were better. That was the trade I actually wanted.

Seeing is not the same as grounding

Choosing a model solved only part of the problem. A meeting video contains an absurd number of possible frames, most of them useless:

• webcam grids
• repeated slides
• cursor movement
• loading states
• transitions
• the same screen held for several minutes

Early multimodal fusion also produced a more dangerous failure. Give the model a transcript plus a dense screen, and it may confidently “read” exact values that are not reliable. A plausible number is worse than no number because it can flow directly into the final note.

The visual pipeline became useful only after I added software guardrails around the model’s judgment:

1. Detect scene changes and candidate frames.
2. Use OCR as a cheap gate to discard visually empty frames.
3. Segment distinct screens so repeated views collapse together.
4. Classify what kind of screen is visible.
5. Ask the model only for details relevant to the surrounding discussion.
6. Preserve timestamps and frame references with the observation.
7. Validate arithmetic when the screen presents real financial data.
8. Treat numbers in design mockups and demos as placeholders unless the evidence says otherwise.

The model reasons about meaning. Software manages selection, provenance, validation, and failure modes.

That division of labor is what turned image descriptions into evidence.

The experiment method that survived

The code changed repeatedly. The experiment loop became surprisingly stable.

Start with a real workload. Toy prompts are useful for checking that a server boots. They are poor predictors of a 57-segment labeling request or a one-hour product review.

Define the ground truth before the run. I kept a known meeting with a set of expected decisions and commitments. Otherwise it is easy to reward a model for producing more words.

Read examples before counting outputs. Three accurate action items beat thirty transcript-shaped fragments. A “better” merge count can hide one enormous, meaningless group.

Change one layer at a time. Model, prompt, parser, serving runtime, frame selection, and synthesis are separate variables. Changing them together produces an impressive demo and very little knowledge.

Keep the negative results. Rejected models, reverted worker pools, and failed prompt patterns are not clutter. They are instructions for the next experimenter—including future me.

Promote a result only after it survives another meeting shape. Standups, design reviews, demos, and external talks reward different behavior. A pipeline tuned to one can hallucinate structure into another.

This made the project slower to declare “done” and much faster to improve.

What Ledger gains from Vision

Without video, Ledger can still create a strong note from the transcript and surrounding context. With Vision, it can be more precise in the moments where language depends on the screen.

The integration follows a fact-ledger model:

• transcript claims remain transcript claims
• visual observations keep their timestamps and frame evidence
• inferred connections stay distinct from direct evidence
• conflicting signals are surfaced instead of quietly blended
• the final note includes only details that improve understanding

This is more important than adding a “screenshots” section. The real gain is that an agent can answer questions such as:

• Which design option did the team actually select?
• What error state was visible when the bug was discussed?
• Did the number on screen match the number people repeated?
• Was a ticket merely mentioned, or was its status visibly changed?

Those answers strengthen the decision log, the action items, and the summary the team will rely on later.

What I would do differently now

I would spend less time asking, “Which model is best?” and more time asking, “Which layer is failing?”

Is the frame irrelevant? Is the prompt asking for the wrong judgment? Did the model return a structure the parser rejected? Is the server applying the configuration I think it is? Did the result get faster only because it did less work? Is the observation supported by the video, or merely compatible with it?

Local LLM work rewards curiosity, but it also rewards suspicion. Every clean-looking output is a hypothesis until it survives contact with the source.

The finished pipeline is useful. The more durable result is the method behind it: real inputs, explicit success criteria, raw-output inspection, reversible experiments, and a sharp boundary between model judgment and software guardrails.

That is how the spare Mac became part of our agent network—and how the meeting notes learned to see.