Jockey by TwelveLabs - The video AI agent that understands your whole library

I'm Aiden, co-founder and CTO at TwelveLabs. Super excited to share Jockey with you today.

Most media search is still metadata search: filenames, timestamps, maybe some object tags from an older computer vision model. None of that captures what's actually happening in your footage: who's in a scene, what they're doing, the context, dialogue, on-screen text. For that you need models that natively understand time and space in video, not a bag of sampled frames.

That's our stack. Marengo, our embedding model, resolves a query like "the moment we almost missed the flight" to real retrieval across video and images, not keyword matching. Pegasus, our video-language model, segments an entire video on a schema you define and returns structured, timestamped moments. Jockey is a unified agentic system that reasons across your videos and images: a reasoning model plus a memory layer that builds a knowledge store from your corpus, so it can decompose a query, retrieve, segment, and reason across the whole thing.

The point is a corpus-level understanding you can act on. Point Jockey at thousands of videos and images, say "cut me a highlight reel" or "pull the best viral moments," and it comes back with timestamped cuts you can use. The model-only approach can't do that as dumping one video into a context window is bound to a single file, and a single forward pass runs out of room fast. It can tell you about one video; it can't reason across your catalog or build a reel from thousands.

Because the models are what we ship and improve continuously, Jockey's reasoning and retrieval quality improve as we push new versions, meaning no re-integration on your end.

Two ways in:

• MCP server: connect Jockey as a tool in Claude and query your library directly. ChatGPT coming soon.

• API: full programmatic access to build custom retrieval or agent workflows on your own library.

This is a research preview, so if you hit edge cases (ambiguous queries, retrieval misses, latency) I want to hear about them. Let us know anytime! 

Best,

Aiden 

Great name Aiden! The "no tags needed" pitch is the part I'd want to/ am going to stress test. Natural language search over video is great for discovery, but for compliance/legal use cases you usually want deterministic, auditable categories, not a model's best guess at a scene. Curious how consistent Pegasus's segmentation is run to run on the same clip.

The compositional queries are where library-scale video search tends to break. Single-concept stuff like 'red car' works fine off embeddings, but 'the moment right after the door opens' needs temporal grounding that flat similarity search can't reach. When we built multimodal search over video the causal and ordering queries were exactly where recall fell off a cliff. Does Jockey's agent decompose those multi-step queries and reason over ordering, or is retrieval a single embedding lookup under the hood?

The "understands your whole media library" angle is really compelling. I already have product screenshots, launch videos, screen recordings, demos, and random clips scattered everywhere, and finding one specific moment usually depends on remembering the filename or roughly when it was created.

Searching by person, moment, or context feels much closer to how people actually remember media. connecting that through MCP is interesting too, because an agent could finally find the exact clip or screenshot needed for a task instead of asking me to dig through folders.. :) Curious how Jockey handles privacy and indexing for very large personal or company libraries.

the corpus-level reasoning is the interesting jump here, most video search tools stop at "find the clip" and Jockey is going further to "reason across everything and build the reel." one thing I'm curious about that's different from the privacy/consistency questions above: as you keep adding new footage to a library over time, does Jockey only need to embed the new additions, or does growing the corpus mean periodically reprocessing the whole thing to keep the reasoning layer coherent?

Was lucky enough to get access to Jockey a week or so ago (thank you team!), and have been blown away at the use cases we've already uncovered. It's changing the way we look at creative strategy across our roster of clients, and we've found some novel ways to extrapolate learnings that are informing some of our performance marketing campaigns that are already showing positive uplift.

Go TwelveLabs!

the split between Marengo for retrieval and Pegasus for schema-based segmentation is a real architecture, not just a wrapper on top of an LLM. one practical question - for personal footage across years of different phones/cameras, is there an upfront processing step you have to run before search works, or does it index incrementally as you add new files to the library?

Congrats on the launch. The whole-library search angle is strong, especially for teams with years of product demos, webinars, and raw footage. When Jockey returns a highlight reel or a moment-based answer, how much provenance does the user get back? I would want every suggestion to point to exact source clips and timestamps so the result is easy to verify before editing or publishing.