Multimodal Video Q&A Engine

Ingestion

Video is an awkward input because the meaning is split across two channels that arrive on different clocks. The pipeline pulls audio with MoviePy and ffmpeg and transcribes it with Whisper at segment-level timestamp granularity, then samples keyframes every two seconds and generates dense visual captions for them in parallel through a bounded thread pool.

Bounded matters. An unbounded pool on a long video will happily open several hundred concurrent captioning requests and get itself rate-limited.

The two streams are fused into a single queryable representation, so a question about something that was said and a question about something that was shown take the same path.

Retrieval without vectors

The default answer here is embeddings: chunk, embed, store in a vector database, retrieve by cosine similarity. That works, and it also means maintaining an index that goes stale, tuning a chunking strategy, and accepting that similarity is only a proxy for relevance.

This uses a two-stage LLM cascade instead. A cheap model does a coarse pass to narrow the candidate set, and a stronger model reasons over what survives: a cost-tiered hierarchy where the expensive model only sees material that has already earned attention. Requests are batched concurrently with asyncio.

The tradeoff is honest: this costs more per query than a vector lookup and would not suit a corpus of millions of documents. For a single video it removes an entire piece of stateful infrastructure and retrieves on meaning rather than proximity.

Interface

Answers stream to the client over server-sent events with real-time token-cost accounting, and every claim carries an inline frame citation that jumps to the moment in the video it came from. Citations are the point. An answer about a video that you cannot verify against the video is just a summary you have to trust.