Jason Kramer, Chief Research Officer
Early in my career, I decided to jump into the dotcom boom—just in time to be a part of the dotcom bust. I joined eCandy, owner of the candy.com domain, at a time when anyone with a familiar-sounding domain name could raise millions. eCandy was pitching candy manufacturers on joining their platform, and the pitch was pure fear: Join us or be left forever in the dustheap of history.
Ultimately, fear didn’t sell, leading to the dotcom bust. Will the AI boom be different?
eCandy was right about one thing: the internet would become a huge and omnipresent influence in our lives. But in the beginning of this revolution, there was a lot of distracting hype to sift through. The same is true of AI and research today. One application that shows great promise: AI Persona Agents.
AI Persona Agents are trained using extensive qualitative and quantitative data to represent consumer and B2B personas. At Vital Findings, we’ve used them to represent physicians, fitness enthusiasts, patients, and young trendsetters, often based on segmentations. We’re big fans, and we think they could change—but not replace—traditional research.
There are a number of companies building dedicated platforms for AI Persona Agents, and most of them are overselling what agents can do. One claims “90% similar to traditional methods,” and another suggests that you use agents “if you’ve ever wished you could get closer to how real people think, feel, and act.” These claims are distorted and misleading. Don’t buy the hype… but at the same time… don’t ignore the potential for real use cases, like pre-testing.
Why are AI Persona Agents so important?
- They’re here and already proving useful—not just hype.
- They improve the front end of research projects, for things like pre-testing surveys, discussion guides, and developing hypotheses to test in research. And they provide a showstopper element when unveiling segmentations.
But:
- They are not a replacement for traditional research—building a good agent requires distilling the essence of a persona from your data, and (ideally) providing edited transcripts the agent can draw on in answering questions.
CASE STUDY: “CAUTIOUS COLLABORATOR” Physician Persona for a MedTech Brand
We recently completed a physician segmentation for a Medtech client, interviewing hundreds of endocrinologists and primary care physicians, conducting dozens of IDIs, and in-person doctor-patient role-play sessions. With such a large-scale effort, the client wanted to make sure the rollout made a big splash and that the segmentation was infused into their research going forward.
To accomplish these goals, we decided to experiment with creating AI Persona Agents. One example persona agent we created was the “Cautious Collaborator,” an endocrinologist segment who is a late adopter of diabetes tech like insulin pumps (“cautious”), works in larger healthcare systems as part of a care team (“collaborator”), and has a patient population that skews toward minorities.
For the segmentation rollout, we pulled out our talking agent (you can literally “call” and talk to them!), and asked the crowd if they had any questions for them. The audience was floored by the results—for example, hearing the “Cautious Collaborator” talk about the challenges her patients face in affording treatment and an emotional story about a patient in the emergency room facing amputation.
After the rollout, we experimented with ways to further embed the segmentation in future research. We discarded the idea of replacing traditional research early on—while the agents’ answers to questions for which we didn’t have training data for made sense, there was no way to validate them without doing further research (in the segmentation rollout, we steered the questions toward topics we knew were included in the transcripts we used for training).
We found that the agents were a terrific addition to research for brainstorming and pre-testing.
Some notable successes:
- AI Agent vs. ChatGPT Pro: We asked the Cautious Collaborator to evaluate surveys and discussion guides and compared that to ChatGPT Pro instructed to act as an endocrinologist. The agent gave more nuanced feedback. For example, it suggested we add socio-economic factors to a response list, while ChatGPT gave general, logic-oriented responses.
- Early-stage Testing. We’ve all faced the issue of testing creative that we intuitively knew wasn’t ready. As a test, we decided to share a physician brochure draft that we knew had bombed in field, causing us to pause the research and revise the stimulus before going back into field. The Cautious Collaborator agent had a strong negative reaction, corroborating the feedback that we heard in field. For future projects, we plan to use the agent to get the ammunition needed to push back before going into field.
- Go Global. We wanted to take our segmentation beyond the US, so we asked our agent what they might look like in the UK, Germany, and France. The results gave us hypotheses we tested in our quant.
Recruiting for tough to reach audiences, like physicians who fit a specific endocrinologist segment, is a challenge that has continually gotten harder—and we need every interview to count. The gold standard is to pre-test our surveys and discussion guides with human respondents, but costs and timing are increasingly making this unrealistic—this is where AI Persona Agents are becoming a key part of the modern research toolkit.
Are AI Agents a Replacement for Research? Not Yet.
There may well be a future where AI Persona Agents can replace some research, particularly low stakes/low value work, but methodologies will need to evolve to develop a class of agents capable of replacing research. For example, a team from Stanford and Google recently parallel-tested AI Persona Agents with real human respondents, finding an 85% correlation in responses… but with a BIG caveat. They generated their persona agents from 1,000 two-hour in-depth IDIs on a long list of topics. Similarly rigorous methodologies will likely be required to develop agents capable of replacing any traditional work.
At the end of the day, the promise of AI Persona Agents is exciting, but the practice needs discipline. If you’re curious about how this could apply to your work—or have a project that might benefit from AI Persona Agents—reach out to me and let’s set up time to talk!
