They Shouldn’t Know It’s a “User” Interview
In a short brainstorming session, we exchanged ideas for the most effective product discovery tools. Sure, the right tool is not right, period. It’s right only in the right context. No silver bullets here, sorry. Still, some evergreens emerge almost every time.
Assumption mapping is one. I personally like user journey mapping, too. However, there’s one thing that we don’t often consider a proper discovery technique, yet it should be a staple. User interviews.
User interviews definitely make it to my top 3. Yet, as with everything, there’s a caveat here.
Subscribe
Before we jump to the actual analysis, I wholeheartedly recommend reading Rob Fitzpatrick’s The Mom Test. It’s short. It’s well-written. It tackles just one idea and does it superbly. Heck, it’s fun. Rarely is there a business book that crosses all the t’s and dots all the i’s. This one does.
Do yourself a favor and get a copy. Better yet, listen to the audiobook. The author does fabulous work turning it into something like an informal chat, rather than a professional yet dry-as-hell narration that’s the baseline for business literature. It feels almost like you got a chance to invite Rob Fitzpatrick for a few hours over coffees or beers. Which I would totally do, if I had a chance.
If you want to skip the reading (don’t), and user interviews are a new concept (they shouldn’t be), here’s the gist of it.
You talk with an existing user of your product (if you have some), or a potential one (if you don’t yet have any), for 30-60 minutes to get valuable insight into your plans. The plans may mean changing UX, adding a new feature, or building an entirely new product.
The tricky part: asking direct, closed questions along the lines of “Do you think this is a good idea?” or “Would you use it if we built it?” is the worst—and utterly useless—way to run user interviews. Also, it’s the most intuitive way, which makes the whole activity tricky.
By contrast, good questions revolve around interviewees’ needs and problems, and how they solve them now. In a way, it’s an exercise in beating around the bush to extract valuable insight that will validate (or, more likely, invalidate) the product assumptions you might have.
When we run engineering internships, we often ask the interns to build an MVP of a simple product. On more than a few occasions, we developed a thing for which we had potential customers inside Lunar. An obvious move was to interview these hypothetical future users.
So there I was, being user-interviewed by two interns. People with relatively limited engineering skills and no experience building products whatsoever. The only preparation they got was some advice from our product folks.
It was one of the best user interviews I’ve ever seen.
I got zero questions leading me to the idea that they might be working on a product. It was all about my problems and how I was handling them. We spent some 45 minutes chatting, and no products, features, or users were ever mentioned.
Lack of experience didn’t hold them back. In fact, it might have helped them. We prompted the interns to explore the problem, and that’s what they did. Founders, on the other hand, tend to be overly focused on their idea, which makes it so much harder to avoid treating their interlocutor as a potential user.
Now that I look back, I think that was the key part. It didn’t feel like I was a user in that interview.
OK, the name of the technique doesn’t do it many favors. It’s called a user interview. It should be something something interview. I don’t know. A problem interview. A need interview. Yeah, I know. You don’t interview a problem.
Anyway. Revolving around the idea of a user tricks us into asking the wrong questions.
Hack 1: Never treat an interviewee as a user. If you mention the product, you’ve lost. If you mention the word “user,” you’ve lost.
Review your questions. If any of them might make an interviewee feel like they are a product user, throw it out and never look back.
Of course, in some contexts it would be blatantly obvious to an interviewee that they are, in fact, a user. It’s fine. Just make sure none of the questions suggest it. In any way.
Our interns run their interview in a pair. In a new environment, trying to prove themselves, most would choose such a setup. It’s less stressful when you’re in it together. You can support each other. It’s a safe play.
Yet, once you get familiar with the task and the environment, it would be tempting to skip the pairing part. In this case, though, it’s so much better to run it as a duo. In this case, 2 \* 1 = 10. User interviews are binary, it seems.
Hack 2: Have two people run a user interview. There are just too many things for one person to handle.
In theory, the interviewer(s) would come with a list of questions, and they’d go through it one by one. In practice, conversation dynamics vary a lot. Some people need but the tiniest hook, and they’ll respond with a lore drop, crossing a third of the questions out of the list instantly. Others need continuous inquiry to give you anything.
The magic, as usual, happens in between the lines. So when one person (Batman) is all focused on navigating the conversation, the other (Robin) should be on the lookout for those hidden gems barely mentioned in passing. In my experience, expecting one person to handle both is too much.
As a bonus, Robin can (should?) be the one to play the “hold your horses” card. When Batman gets carried away, assumes they talk with a user, and starts asking product questions (which will inevitably happen), the other person is in the best position to notice. And react.
In the past, one reason to have another interviewer when talking to a potential customer was to have a note taker. There probably are people on this planet who can actively guide the conversation while taking notes. All three of them still make their interlocutor lose focus when they switch between the tasks.
“But wait! We have AI! AI can generate the transcription and the summary of an interview perfectly, no sweat!”
Yes. And no. Yes, AI can generate the transcription well enough. So what? Do you have product people in your team who will be reading these transcripts? I mean, seriously, do you?
The time needed to read and understand the transcript is comparable to how much it took to chat. So you might as well do the thing as it is happening.
Also, don’t expect AI to extract genuinely original insight. Novel observations happen at the intersection of the thoughts and experiences of the interviewee and the interviewers. Claude has access to neither.
Trying to work with an AI notetaker is a direct equivalent of telling AI to generate a dozen pages of slop based on some sources. We’d rather see the prompt, not the prompt output.
Hack 3: AI notetaking is not a replacement for the second interviewer. If anything, it adds work, not reduces it.
A competent human will intuitively understand which parts of the conversation are valuable and which are not. LLMs, by definition, do not have “intuitive understanding.”
Let me redefine what a user interview is. It’s spending 45-60 minutes, talking to a person who might never even be interested in your product, to harvest between 0 and 3 short insights. It doesn’t sound like an effective use of time.
That is, unless one of these insights is valuable enough to justify investment. And it just so happens that every now and then, they are. Even if none of them are on their own, if you gather enough data points, you’ll start seeing patterns. That’s value, too.
So you either have enough reflex to see those on the spot, or you rely on synthesizing the interviews afterwards.
Hack 4: Don’t outsource the synthesis. AI on its own will produce very shallow output.
That’s, in fact, my old pet peeve with AI creativity. AI models will get only as creative as we make them. And we need expert knowledge to make it happen. Put differently (in Teresa Torres’ words):
“I can’t say AI is great at synthesis. AI is great at synthesis if you’re great at synthesis and you know how to teach AI how to be good at synthesis.”
The reason, again, following Teresa Torres, is that “there are lots of ways to synthesize an interview.” Claude or NotebookLM can’t know on its own what’s contextually the most valuable vantage point. It can’t pull ad hoc all the experience and intuition of a product person.
Ultimately, synthesizing insight is the most valuable part of a user interview. In general, we don’t outsource the core of our businesses. Why should we do it for the discovery process, then?
A frequent bit of advice for user interviews is to finish by asking: “What question should I have asked but didn’t?”
Don’t overestimate it.
In theory, it does the trick. It’s an open-ended question that probes the territory you didn’t explore. In practice, it suffers from the very same pattern that plagues the “Would you use my product?” inquiry.
Rob Fitzpatrick frames it as a fatal mistake (emphasis mine):
“Most of your meetings will end with a compliment. It feels good. They said they liked it! Unfortunately, they’re almost certainly lying. Not necessarily intentionally.”
By the same token, when asked the “ultimate” question, we will invent stuff the interviewer wants to hear. “What question should I have asked but didn’t?” Wait, there is supposed to be something more? Let me make something up.
Hack 5: No single question is pivotal. What matters happens in between the lines.
No specific script will land you good insight. It’s all about navigating the conversation ad hoc, depending on what you’re getting thus far. Ignore that, and the answer to the “silver bullet” question may be either golden or completely misleading. Or anything in between.
Finally, no matter how you try, there’s no way to avoid bias. I mean, we go in there with a hypothesis and a hope. It’s not like we’re neutral. “I’m indifferent to whether this idea is good or bad,” said no founder ever.
No wonder. The list of biases we need to overcome in this situation is impressive:
- Confirmation bias makes us ignore counterevidence to the hypothesis.
- Survivorship bias makes us overly optimistic about the odds.
- Social desirability bias makes interviewees say things we’d like to hear.
- Self-selection bias makes the pool of people we talk to more likely to be interested in the thing.
- Say-do gap makes people state intentions they wouldn’t eventually act on.
- Overconfidence bias makes us overestimate the weight of our own interpretations.
- Observer-expectancy effect makes us directly influence the answers we get.
- Illusion of validity makes us overestimate our ability to predict outcomes based on data.
Should I go further?
Hack 6: Assume the confirmation signal is much weaker than it looks. Set the threshold high or cut the perceived strength of the signal by half.
No matter how aware we are of biases, it’s of little help. We believe knowing of a bias helps, but it does not. And yes, there is a bias for that, too. It’s called bias blind spot.
In Thinking, Fast and Slow, Daniel Kahneman argues for setting verifiable criteria beforehand and sticking to them, and them only. In the context of qualitative techniques, such as user interviews, it will prove challenging.
Still, it’s a safe bet to assume the positive signal is always weaker than it looks, while the negative is always stronger.
When I started writing this article, I was like, “Hey, let’s write a short, lightweight piece on some user interview caveats.” Some 2000 words later, it feels like I could keep going.
Yeah, Rob Fitzpatrick took well over 30k words to write The Mom Test, which is short and lightweight. And it covers the same narrow topic. What have I been thinking?
Looking back at it, though, I’m like “Scratch that. If there’s only one thing I’d like you to remember, it’s none of the above.”
As innocent as user interviews look, it’s so damn easy to hurt yourself running them. If you take assumption mapping or user journey mapping and do them wrong, the worst thing that can happen is you waste some time. The outcome will be meh, but it will hardly push you in the wrong direction.
If you do user interviews wrong, they’ll absolutely push you in the wrong direction. It’s a “great if done well, harmful if not” kind of technique.
---
More posts on product discovery techniques:
[](https://pawelbrodzinski.substack.com/p/my-favorite-product-discovery-tool)
My Favorite Product Discovery Tool: Assumption Mapping
[](https://pawelbrodzinski.substack.com/p/my-favorite-product-discovery-tool)
[](https://pawelbrodzinski.substack.com/p/my-favorite-product-discovery-tool)Pawel Brodzinski
·
Jul 8
[](https://pawelbrodzinski.substack.com/p/user-journey-mapping-we-want-it-to)
User Journey Mapping: We Want It to Be Shit
[](https://pawelbrodzinski.substack.com/p/user-journey-mapping-we-want-it-to)
[](https://pawelbrodzinski.substack.com/p/user-journey-mapping-we-want-it-to)Pawel Brodzinski
·
Jul 22
[](https://pawelbrodzinski.substack.com/p/user-interviews-with-ai-models-useful)
User Interviews with AI Models: Useful, But Not a Gamechanger
[](https://pawelbrodzinski.substack.com/p/user-interviews-with-ai-models-useful)
[](https://pawelbrodzinski.substack.com/p/user-interviews-with-ai-models-useful)Pawel Brodzinski
·
Apr 1
---
Thanks for reading The Pre-Pre-Seed! I appreciate sharing or subscribing. That is, given that you like my work.
Subscribe
---
I’m typing these things by hand. Like an animal. 웃 <https://okhuman.com/jcttRg>