Synthetic Users for Everyday Usability Testing

Brent started his demo with a question for the room of builders. "How many people have I coded an iOS app? Raise your hand."

"Quite a few. Quite a few. Cool," he said, reading the room. The setting was informal, a live look at a project he described as "very raw" and just a couple of weeks old. He didn't even have a name for it yet. This was a look at a new tool, and he set the room's expectations with a builder's candor. "I'm going to give it a live demonstration. And it's probably not going to work."

Catching Silly Mistakes Before Real Users

Brent's goal is to use AI personas for what he calls "synthetic usability testing." The idea isn't to replace human feedback, but to catch obvious friction points before an app gets to a real person. "I think there's also an opportunity to do usability testing and try to get those at least low hanging fruit of kind of silly mistakes that might be made when you're coding an app," he explained.

Many teams already use agents for functional testing, making sure the app works correctly. Brent is interested in the next step: finding out if the app is usable. An automated first pass can find the simple flaws that creators, too close to their own work, often miss. Once the synthetic user has cleared the most basic issues, the app is more ready for nuanced feedback from real users.

A Cat Named Whiskers Gets a Profile

To demonstrate, Brent ran his Mac application against a simple iOS cat-sitting app. He gave the synthetic user a task: "set up the morning feeding hours for the cat of the friend who can cat sit." He configured the AI persona as someone new to this kind of app, a bit hesitant with tech, and in a hurry.

The AI analyzed the screen and chose to proceed as a guest, skipping the account creation step. It navigated to the cat profile section and gave the cat a name. "Every time it usually chooses whiskers," Brent noted. The AI typed the name, created the profile, and moved on to setting the feeding schedule. At the end of the run, the tool produced a report detailing how long the task took, the number of actions, and key moments worth reviewing.

But it wasn't a clean success. When it came time for the AI to answer debrief questions about the experience, Brent was frank. "It did fail," he said. "Some of the tech is not quite great... we're still doing that very, very early." That failure was the point. This was not a polished demo, but a real look at a work in progress.

Brent's demo puts a sharp point on the difference between building something that is functional and building something that is usable, a distinction our most effective builders learn is the whole game. Choosing to proceed as a guest is exactly the kind of unexamined friction point that can lose a new user. We want them to sign up, so we design for that. We forget how many people just want to see if the app is useful first. An automated tool that finds that single moment of hesitation is a powerful asset.

Before moving to Q&A, Brent made a direct appeal to the room of builders. "We'd love feedback on this concept," he said. "And if we have any apps that you think would be worth testing... try and test them."

Calibrating the Synthetic User

The audience questions dug into the core challenge: calibration. One person asked if the personas could eventually be shaped by real user data, "so the personas become more representative of your actual user base."

Brent confirmed this was a critical part of the roadmap. "One thing we haven't gone into yet," he said, "is doing like a real usability test of real users versus synthetic users. We need to check and calibrate to see how that performs." The bigger opportunity, he explained, is not just to mimic users but to give builders a new skill. "We can put in a lot of usability testing expertise. Somebody new that creating an app wouldn't know." The goal is to encode an expert's eye for common problems into a tool anyone can use.

Another audience member followed up, suggesting a future where you could "rent somebody else for someone who is very well-calibrated." Brent smiled. "So we have a persona star. Yeah, maybe."

For now, the project is still in its raw, early stages. But the open question it raises is a practical one for any builder. Can a synthetic user, running through a simple task, reliably find the "silly mistakes" and obvious friction points before you put your work in front of another person? Watching this tool develop is watching an answer to that question take shape.

---

SQ Collective hosts Coworking Fridays for founders, operators, and AI builders working through real product questions in Singapore.

Join an upcoming Coworking Friday: https://lu.ma/ai-labs

Explore SQ Collective: https://www.sq-collective.com

Michael