By Jason Kramer, Chief Research Officer, Vital Findings
At Vital Findings, we’ve been sifting through the massive volume of new AI tools so you can focus on adding value to your business. Today, we’re reviewing AI open ends: what they are, what they’re good for, and when to use them.
In the past, open ends have been the primary way qualitative methods have been used in quantitative research. The challenge we’ve all faced is the lack of depth in these responses. We often find ourselves using open ends as much to determine the quality of respondents as to gather actual feedback. Increasingly, we’ve been adding AI-generated probes to our open ends to make them useful again.
What is an AI open end?
With an AI open end, we first program a typical open end. For example, in a study about a new running shoe, we asked, “What do you like most about this shoe?” For a respondent who said, “It looks comfortable,” the AI follow-up might ask, “Comfort is certainly important. What about this shoe makes you think it will be comfortable?”
Some recent examples of how we’ve used AI Open Ends:
- Enhancing key metrics: In a brand study where “is a leader in the industry” is a key metric, we asked respondents to explain their rating of the brand and added an AI follow-up to dive deeper into this metric. In the past, we would have had to ask a closed-ended question to get enough information to report.
- Expanding a segmentation map: In a follow-up to a segmentation study, we wanted to know the extent to which the segmentation could apply to other countries. We presented respondents with their algorithm-assigned segment and then asked whether they agreed or disagreed with this assessment. AI then probed to get richer insights into what drove their opinion.
- Product Insight: In a recent study with physicians, we used an AI open end to probe deeper into how a new product could change prescription trends and why.
What we like about AI open ends:
- AI follow-ups can spur respondents to write more.
- If a respondent writes gibberish, it will ask them to fix it.
- AI open ends improve the experience for respondents by acknowledging what a respondent says and responding dynamically.
- We typically get 40-50% more words from an open end with an AI follow-up than in an open end without AI.
Cons of using AI open ends:
- An AI follow-up generates an additional question, so you need to plan in advance for it in terms of survey length.
- Most survey platforms have yet to implement this functionality, and typically require a specialized add-on (at an additional cost).
- Generally, the AI decides what to probe on, but you can direct it with some extra effort.
- Analytically, we have to decide whether to analyze both open ends together or separately (we typically find that they should be analyzed together).
When should you consider an AI open end?
AI open ends will increasingly become standard practice, and the tipping point for this will likely be when the most widely used survey platforms add this functionality.
However, we need to carefully consider what questions could really benefit from them. Using an AI follow-up just to get more words out of respondents can backfire, especially if more than one AI probe is included. These questions can start to feel repetitive, like the AI probe is asking the same question using different words.
When we think about where to use AI open-end probing in our surveys, we have narrowed it down to three question types:
- More complex or exploratory questions (vs. straightforward questions)
- Adding depth and nuance to hero stats (i.e., why did you say this type of TV content is your favorite?)
- Defending their answer (i.e., why did you say this brand is a leader in the category?)
AI open ends are a great tool that can add a significantly higher level of insight for quant surveys, but to get the most out of it, you need a research team experienced in how to get the most value out of them.
