Launched this week
Overmind
Open-source platform for continuous learning
561 followers
Open-source platform for continuous learning
561 followers
Overmind is the model training platform for AI teams. Turn your production traces into specialised models you own, trained, benchmarked, and served.
Follow you guys for some time and this one looks crazy! how do you know the small model's good enough to swap in for what I’m using right now? do I guess right you test it against the current one?
Spot on, Art! Every small model we train gets benchmarked against the model you run today, on the exact same frozen set of your own production traces, scored on the evals you defined for what "good" means. You see the before/after numbers side by side, so the swap only happens when the small model actually wins. And when it does not, Overmind tells you that too, instead of letting you ship a coin flip :D Easiest way to see it is on your own agent: one prompt in your coding agent, "Set me up on Overmind https://console.overmindlab.ai/setup.md".
Thank you for following along all this time 🙌
The versioned eval sets are a useful part of this. I'd want to be able to replay the same cases when a new model starts behaving differently.
Hey @oleksii_sekundant 👋 That's exactly what we built for. You turn your real production traffic into a test set and freeze it. Then when a new model comes along (a frontier one or one you fine-tuned with us), you replay the same cases on it next to your current model. You see what got better and what broke before your users do. We'd love for you to point Overmind at one of your agents and tell us how it feels.
@oleksii_sekundant adding to frey's answer: the eval set is pinned to a dataset version, so when you replay it on a new model you're comparing on the exact same rows, not "roughly the same traffic". we also split by conversation rather than by row, so a model can't score well just because it saw other turns of the same chat in training
The pinned dataset version detail stood out to me — replaying the exact same rows, split by conversation rather than by row, is what would make me trust a before/after comparison. The Qwen 0.8B example going from 36.7% to 84% on message classification is striking. How do you guard against the eval set going stale as the agent's tools or prompts change underneath it? Do you version the tool and prompt snapshot alongside the dataset, or flag cases that no longer match the current agent setup?
@jcbuilds Great question JC, and the answer is both! It all happens inside your coding agent, so the same agent that edits your prompts or tools also has the diff and the eval set in front of it.
Versioned with the data: every eval row keeps the exact system prompt, tool schemas and recorded tool results it ran with. Dataset versions are immutable and fingerprinted, every run pins the version it used, and we record the git commit each part of the agent came from.
Flagged when it drifts: we check each version row by row against the agent’s current setup (changed prompt, tools it no longer declares, missing inputs). Your coding agent sees something like “120 of 900 rows do not match” right next to the diff that caused it, and can keep those rows as a regression set or cut a new version. Nothing gets rewritten silently.
Caught at replay: evals replay tool calls from the recorded results. If the model calls a tool in a way we have no recording for, that row is flagged and the run is not marked as trusted.
Would love to hear how Haplo handles this today 🙌
As a marketer, I'm not new to using AI, but I am new to actually building and running AI agents. Overmind seems like it would be able to help me get there much quicker than I anticipated, and without hiring an entire ML team, too 😁 Best of luck with the launch!
@viktoriia_kononenko Thanks Viktoriia! You don't need an ML team for this! It's as easy as pasting a single prompt in your coding agent (Claude Code, Cursor, Codex), and it does the setup and training for you.
Set me up on Overmind https://console.overmindlab.ai/setup.md
My DMs are always open if you want any further help :)
@viktoriia_kononenko Love this, Viktoriia! Marketers are going to be some of the best agent builders, because you already have a decent idea of what "good" looks like for your audience. That is basically half the work of training a model, and Overmind handles the rest. Would love to hear what your first agent ends up doing 😁
hey PH, pritam here, one of the founding engineers at overmind 👋
the part i've been living in is fine-tuning and evals. the thing that surprised me most building it: the eval is what makes the training worth anything. we had a run recently where every model, big and small, scored ~67% on a yes/no task. no amount of tuning fixes that, the data had no signal. overmind showed it in the before/after numbers instead of letting us ship a "fine-tuned" model that was really a coin flip.
the happy path looks more like our support demo: qwen 3.5 0.8b went from 36.7% to 84.0% on messages it had never seen, in 23 minutes from launch to live.
if you've got an agent in prod making the same kind of call over and over (routing, classification, tool choice), that's where i'd start. ask me anything about how the evals work
@pritamsonihsr The eval wizard has spoken 🧙♂️🙌
The idea of turning production traces into a model for a specific task is interesting. Teams already have a lot to learn from how their agents perform in the real world. Congrats on the launch!
@marianna_tymchuk Thank you Marianna! Your agents are already telling you what good looks like, we just help you listen 😄
@marianna_tymchuk Thank you Marianna!! Yes how your agents behave in the real world is the most data you've got, treasure it, and use it to train more efficient smaller models! Simply start from your own data and one prompt in whatever coding agent you're using. Are you running any agents in Stripo atm?
A big shoutout to the guy who designed the website ❤️
@vijayanandkailash our second proudest asset (after the platform) :)
@vijayanandkailash The brick-breaker easter egg is my favourite! https://www.newgenre.studio/ and we had a blast designing our website!
@vijayanandkailash brand is the new moat! Totally agree.
@vijayanandkailash @hissingkitty brand + the data flywheel of proprietary feedback compounding with usage, captured by agent training platforms - agreed!! ;)