TheySaid 4.0 - See how users will react, right where you build.
by•
Put what you’re building in front of real customers or synthetic users trained on them when you need an answer in minutes without leaving Claude, Cursor, or Jira. Nothing ships on a guess.
Replies
@chrishicken This is looks useful! Can it help small teams test Ideas without spending hours on user research?
@rohit62661 yes, small teams are honestly who we had in mind. the slow part of research is usually recruiting and scheduling, not the testing itself. here you point it at a url or a prototype, the AI drafts the tasks and questions, you approve, and a couple of AI testers click through it while you keep working. most sessions we've run in the last month came back in roughly 5 to 18 minutes. you can do the same with real people too when you want that extra confidence. what kind of idea are you trying to test?
Yes @rohit62661! You can get real feedback without blocking off a week for research. The 10 free synthetic tests are a great place to start.
@chrishicken @rohit62661 I'd argue that if you're a small team, you actually can't afford NOT to use AI-powered research tools! 😂
Small teams move fast because they have to, but moving fast in the wrong direction is expensive. Too often, teams skip validation because they don't have the time, budget, or headcount, and end up building something customers don't want.
Now you can use AI to recruit real users, test with synthetic users, and validate ideas while you're building. Build fast, but build the RIGHT thing.
With speed and cost no longer being major barriers, there's really no excuse to skip validation anymore!
@rohit62661 +1 to everything already mentioned. I would add that you'll get the most savings if you use MCP and launch your tests from wherever you're already building. For example if you go to Claude --> Settings --> Connectors, you'll see TheySaid listed. Once connected you can launch a test on something you're building in Claude, or anything else.
@tadej_jevsevar Thank you so much, Really appreciate the support!
@tadej_jevsevar Thank you!
@tadej_jevsevar Thank you !!
@tadej_jevsevar Appreciate your support!
Hi, team congrats. very curious how Synthetic users get created? is there any metrics showing the accuracy of simulation?
@gavin_fgg0815 thanks Gavin! creation happens in three passes. first who they are (role, company, demographics, voice), then how they think, around 19 traits like patience, trust in tech, how direct they are and who they blame when something breaks, then how they behave and talk. you can describe the audience yourself or build testers straight from real people's sessions. then each one goes through your product in a real browser, thinking out loud while the screen records. on accuracy, we ran 25 real shoppers vs 25 AI testers each built from one of those people, same live amazon task, and 21 of 25 ended up with the same final choice as their human, with the overall split and confidence very close too. the white paper on our launch page has the full numbers
Great questions @gavin_fgg0815! Arnab covered how they’re built, so I’ll just add this: the best accuracy metric is your own product. What would you want to test?
@gavin_fgg0815 you can read a white paper we just published on this topic here:
https://www.theysaid.io/human-vs-synthetic-testers
Note: accuracy improves with context and data. Out of the box results will vary, but as you add more context and data for AI to work with, you can dramatically improve the accuracy of simulation.
Looks promising! I'd definitely use that to get some feedback on users flows. Good job team!
@elie_molzino Thanks! User flows are the perfect place to start. Point it at a flow you’re unsure about and see where testers get stuck. Curious - how you are getting feedback on your user flows today?
@elie_molzino thanks Elie! one tip for flows, set the tester to phone mode too if your users are mostly on mobile, it runs the same flow as a touch phone user and the stuck points are often totally different from desktop
@elie_molzino the old way of thinking about this is doing it on a project-by-project basis. The way we've set it up, validation becomes infrastructure. Incorporated into how you build, so you never ship another product without validating it with users. Using our connectors and MCP, you can connect your important tools:
Design -> Claude Design, v0, Replit, Bolt, Lovable
Coding -> Claude Code, Codex, Github Co-pilot
Ticketing --> Jira, ClickUp, Monday, Linear
The part I keep coming back to is the audience answering and asking questions as if they are talking to you, after training in under a minute from a URL or internal doc. Do the questions the synthetic users ask back get surfaced separately from the usability feedback? That feels like the more useful signal to me — a tester asking about something missing usually points at a gap in the doc or the onboarding itself, rather than just a confusing screen. And when a question goes beyond what was in the URL or doc, does the tester say so, or does it answer from general knowledge?
@jcbuilds good catch, those are actually two different pieces. the URL or doc training is for the AI moderator, it runs the interview and when a real respondent asks something mid conversation it answers from your docs, short and in their language, then gets back to your questions. the synthetic testers are separate, they're the ones clicking through your product. today the questions people ask back live in the conversation transcript next to their answers, so you can read every one in context. and i agree with you, a respondent asking about something that isn't in your docs is a really useful signal for gaps in onboarding, it's a good nudge for us to pull those out more explicitly
@jcbuilds Arnab summed it up nicely! If you want to see every question people ask during interviews, you can create a Skill in TheySaid. Skills let you run your own analysis on your data. Set one up to pull every question respondents asked across your transcripts into one list, then run it on each project, and you’ve got a running list of gaps in your docs or onboarding.
@jcbuilds One last thing that's important to add. You get to determine how much creativity and knowledge each synthetic tester is allowed to have. If you limit the knowledge of the tester to just a document, then the tester will not be able come up with a creative answer. However, if you give the Synthetic Audience a broader knowledge, then you'll see more creative answers.
You can take it to the next level by providing us with additional data and context to build the Synthetic Audience. Then your Synthetic Audiences will have full freedom to take action within the boundaries of the data provided.
The term validation debt fits my team to a T because we work quickly but then never do the step of validating if it worked.
@hamza_aafzal haha you're not alone, it's the step that always gets cut when the sprint gets tight. that's exactly why we made it trigger on its own, move a ticket to design done in jira, linear, clickup, asana or monday and the testers run and post results back in the ticket, nobody has to remember. what's your team shipping on right now?
saw in the reviews that the voice mode gets people way more honest than a form, makes sense. how close do the synthetic users get to the real customers they’re trained on?
@aurelia_zambon on this page you can download a white paper we published on the results:
https://www.theysaid.io/v4
TLDR; Humans and Synthetic overlap between 85-90% of the time, once synthetic is trained on human data.
Having said that, you should just launch a test on your own. 20 humans vs 20 synthetic, and see the results for yourself!
@aurelia_zambon Adding one thing to Chris's point: the "trained on human data" part matters a lot. In our Dyson vs Shark study, each AI tester was built from one real shopper's session (their motivation, worldview and what they already knew going in). The biggest difference wasn't what they chose, it was how they shopped. AI testers were very diligent: more products compared, deeper into the reviews. Humans skimmed stars and stopped at "good enough." So synthetic gets you the direction fast, and real people add the taste and emotion. Running both side by side, like Chris said, is the best way to see it on your own product.
@chrishickenThank you for your reply. I've worked extensively with tools like lyssna.com, so I'm really interested in this approach to user research.
@aurelia_zambon oh nice, then you'll probably spot the differences fast. the way i think about it synthetic isn't a replacement for the human round, it's the first pass. AI testers click through the flow in a real browser and talk out loud while the screen records, so the obvious stuff gets fixed before real people ever see it. what kind of studies were you mostly running on lyssna?
@aurelia_zambon I've heard good things about their team. I was the 4th hire at UserTesting, and COO there for 8 years. So I'm as "OG" as it comes in this category, and I have a pretty good sense of where the category is headed. We've been building our synthetic technology for over a year ... it's WAY harder to make AI behave and think like humans than we expected.
When you try it out, let me know what you think.
@aurelia_zambon Great question! We've actually been running side-by-side studies to measure exactly that!
We start with real humans, record their behavior, reasoning, and decisions, then create synthetic twins and give them the same tasks.
In our Shark vs. Dyson study, 84% of synthetic twins made the same brand choice as their human counterparts (21 out of 25). Pretty strong alignment!
What we've learned is that replicating usability behavior and product comparisons is relatively straightforward. Personal preferences and emotional decisions are trickier, but we've made a lot of progress through repeated studies. We're now working on our next feature, reusable digital twins, to take this even further.
The goal isn't to replace humans, but to learn from real customers and bring their perspectives into future product decisions without recruiting them every single time.
We published the full study, including where AI performed well and where it didn't: https://www.theysaid.io/human-vs-synthetic-testers
Having a ticket finishing design in Jira or Linear trigger a synthetic test on its own is a handy idea. how do you keep the synthetic users from just agreeing with everything?
@neerajnathany Great question! Synthetic testers in TheySaid aren’t generic “yes-bots.” You can build them from your real human testers’ responses, so each one picks up how an actual customer thinks, decides, and reacts. You can also shape an audience yourself: their goals and motivation, their worldview (trusting or suspicious, optimistic or cynical), and their knowledge base. They also have to complete real tasks, so you see exactly where they hesitate or get stuck. In our Dyson vs Shark study, the AI testers were actually tougher than the humans.
@neerajnathany adding the engineering side to Jaala's answer, this was a real problem for us early on. we measured one project where 18 of 22 generated testers came out as basically the same polite 44 year old ops director, so we rebuilt how the batch gets created. now each tester gets its own mix of traits like how people pleasing they are, how direct they are, how patient, and who they blame when something breaks, and those actually change how they scroll, type and critique. a low people pleasing, low patience tester will just tell you the flow is annoying and bail
@neerajnathany I'll add that we've done extensive work to get AI to think and act like a human. You'll be pleasantly surprised by the types of critiques and problems synthetic encounters. They'll even miss things when scanning a page, just like a human.
@chrishicken good to know they skim and miss things like a person would. do you let people pick how distracted or rushed a tester is for a given run?
@neerajnathany partly, yes. on the audience you get ranges for things like patience, how they read (skim vs read every word), how much they care about finishing the task and how picky they are, so a low patience, skimming range gets you a pretty rushed tester. the mood itself, hurried, tired, frustrated, and things like being easily distracted get drawn per tester inside those ranges so a batch doesn't all behave the same, you don't pin that one directly yet
I'd be interested in the cases where real people and synthetic testers disagree. Those differences sound useful for deciding which assumptions need another look.
@oleksii_sekundant yes, that's honestly my favorite part of the data. in our dyson vs shark study the AI testers kept pushing for hard proof, battery runtime per power mode, actual suction numbers, warranty length. came up in about 7 sessions and most humans just scrolled past it. for a brand that's a pretty clear signal the product page is leaning on vague claims. the humans on the other side brought stuff like strong opinions about dyson or shark from past experience and caring about the color. since synthetic and human responses sit in the same study with a filter, you can line them up and see exactly where they split
@oleksii_sekundant Love this question! The disagreements are often where the most useful insights come from.
@oleksii_sekundant the results are different from what you expect. Here's what happens in almost every head to head study with synthetic vs human audiences:
Humans and Synthetic Identify about 80% of the same issues
Synthetic identifies 10% of issues that humans don't
Humans identify 10% of issues that synthetic don't
So you synthetic audiences end up being a new source of feedback not available previously.
@oleksii_sekundant Great point! What we've found is that it's less about humans and AI disagreeing, and more about how much each one uncovers.
There's a lot of overlap in their findings, but AI can sometimes catch things humans miss, and vice versa. Humans bring real experiences, emotions, and personal context, while AI can explore more paths, dig deeper, and uncover additional issues.
That's what excites me most! It's not human vs. AI, it's human + AI. You get richer insights, broader coverage, and faster validation than relying on either one alone.
We actually published a study comparing the two: https://www.theysaid.io/human-vs-synthetic-testers
Getting feedback on a half finished screen without leaving Cursor is where I would actually use this, since setting up a proper usability test never survives contact with a normal Tuesday.
@karimbenkeroum Haha, Product Hunt says “Likely AI” next to your comment. Did your AI agent help you write this? 😂 Either way, you nailed the use case! Getting feedback while you're still building, without interrupting your workflow, is exactly what we're going for. Thanks for the support, Kareem! 🙌
@karimbenkeroum MCP for the win! Testing while you’re still building is exactly what we built this for. You can make testing part of your workflow without it becoming a full-time job
@karimbenkeroum the half finished screen case is actually the one we had to engineer around the most. a lot of in progress work lives behind a login, private previews and design links the AI testers can't open, so when there's no public url the agent can bundle the screen into a single html snapshot and send that over for the testers to run against. you also get a quick first pass back in seconds while the deeper session runs in the background, so you can keep going with your tuesday
@karimbenkeroum +1 to Jaala's comment about MCP. No need to log into another tool to do your product/concept validation.