Let’s get one thing straight: as a product manager, you are biased, and this affects the outcome of your user interviews.

Let’s get another thing straight: the person you’re interviewing is also biased. This is the inherent folly of all user interviews.

It’s no one’s fault – human nature predisposes us to adjust our answers based on our audience and the way we want people to perceive us.

To avoid bias on both sides of the conversation, it’s important to structure your user interviews in a way that intentionally reduces your ability to influence your respondent’s answers, as well as their tendency to try to give you the ‘right’ answer.

Small but important adjustments to your phrasing ensure you can glean more transparent, accurate, and contextualized insights from respondents.

Limiting bias

The first step in reducing the likelihood of getting biased feedback is to understand the reasoning behind these biases.

Product managers

First, turn the lens on yourself. Your bias as a product manager, hopefully, is not that strong. Most product managers try to bring a truly open mind to every user interview and are focused on hearing that users’ specific experience, habits, and pain points.

However, you may be inadvertently asking leading questions. They seem open, but they subtly plant ideas in the user’s head that limit the scope of what they think they can say in response.

Author and product management coach Teresa Torres highlights the subtle difference between a leading question and a truly open question with this example:

“What’s the difference between asking, ‘What flavor do you want – strawberry? Vanilla? Chocolate?’ and ‘What flavor do you want?’

In response to the first question, the respondent is more likely to pick their favorite of the three options you gave them. They may like vanilla better than chocolate and strawberry, but in truth, they really wanted raspberry sorbet.

The second question keeps all options on the table. It doesn’t suggest, it doesn’t plant ideas in the respondent’s head. The flavor they ask for will probably be the one they truly like best.

Your respondent

In most cases, the person you’re interviewing is more biased than you by virtue of being the one in the hot seat.

We all want to be liked, we all want to be helpful when we can, and we all want to come across as the best version of ourselves. Particularly in professional settings, most people tend to answer questions in a way that reflects themselves in a positive light.

These biases are especially obvious when product managers ask respondents to 1.) judge their own behaviors, 2.) predict how they will behave in the future, or 3.) estimate how much they would be willing to invest in your solution.

Self-judgment

Let’s discuss an instance where a question asking respondents to judge their own behaviors may lead to a biased answer.

In Classify’s case, we’re developing an app that helps people stay organized and more easily prepare for meetings.

Say we asked, “Do you have trouble staying organized?”

The respondent may not want to admit they’re disorganized, or they may want to downplay how much they struggle to stay on top of things.

Being disorganized has negative connotations, and product managers in particular are expected to stay organized as part of their role. This kind of question could lead to a biased answer due to the respondent wanting to be perceived in a positive light.

Instead of asking whether someone needs to improve the way they work or resolve a bad habit, it’s better to ask an open-ended question about their specific actions.

For example: “How do you organize your information?”

This does not impose a judgment on whether the respondent is organized or disorganized – it simply asks that they explain their own process and the actions they take to keep track of information.

Speculation

The second category of questions prompts respondents to predict how they will behave in the future.

An example would be if our product manager asked, “How much time would you want to block off for deep work next week?”

This is a mistake, according to Torres.

“You don’t want to ask people about what they would do,” she explained in a blog post. “We are notoriously bad at predicting our future behavior.”

“We believe we will have more time in the future,” she continued. “We don’t ask what could go wrong or what could keep us from doing what we intend.”

Furthermore, we often project an idealized version of ourselves when talking about what we’ll do in the future. We all want to be the kind of person that goes to the gym, or eats healthy on weekdays, or meticulously organizes our files and information throughout the work day.

But the truth is that life is messy, and we rarely do all the things we intend to do in a day.

Rather than asking about how the person that you’re interviewing will behave in the future, it’s better to ask what they have done in the past.  

Example: “Have you ever tried to block off time to do deep work?”

It’s also useful to ask for specific stories and examples of things they’ve done to gain insight into the circumstances that drove certain actions, as well as  the feelings associated with these actions.  

This requires only a slight adjustment to the previous question: “Talk about a time you blocked off a few hours to engage in deep work.”

This gives you context for their behavior and illuminates which actions make them feel good, which actions induce stress, etc.

Future time/money investment

Asking someone for an estimate about how much they’re willing to pay for a specific service or app rarely yields an accurate answer.

If we asked: “Would you pay $9.99 a month for a tool that automatically organizes your information across apps and gives you context for every meeting and project in your calendar?”

We’d receive a lot of optimistic responses.

That’s due in large part to the fact that people overestimate how much money they will make in the future and how much time they will have to use your product, Torres explained. Also, she said, “they genuinely want to be nice and tell you yes.”

Our clouded perceptions of how we will spend our own money and time paired with our tendency to people-please leads many product managers and founders to come away from a user interview thinking there are a lot more willing customers for their product than actually exist.

This becomes a problem when trying to find product-market fit – we estimate there will be a large pool of excited users, when in fact many of the people who said they would be thrilled to use our app were simply telling us what we wanted to hear.

To limit bias and get a more accurate picture of the behaviors, needs, and size of a particular segment of the market, it’s necessary to avoid these kinds of questions and derive answers through a less direct, more open line of questioning.

A better question would be: “Do you pay for Superhuman?”

This is an app similarly focused on helping people stay organized so they can do their best work.

Asking someone if they’ve paid for a similar service in the past rather than asking if they would pay for your service in the future removes speculation and roots their feedback in their own actions.

Structuring an unbiased user interview

Now that we understand the reasoning behind our biases and what not to say, let’s go over the kinds of questions that will yield transparent, reliable feedback.

We’ll use examples from Classify’s own user interviews.

Step 1: Categorizing your respondent

Start off by getting a little context on the person you’re talking to.

• Talk a bit about what you do
• How big is your team?
  

These are standard introductory questions about the scope of the respondent’s responsibilities. Their answers will allow you to flesh out the profiles of their particular user persona and categorize their company alongside others you’ve worked with.

Step 2: Assessing the user’s current pain points

• What’s a typical day?
• Do you need to prepare for meetings?
• Which meetings require the most prep?
• Any challenges in that respect?
• How do you keep your files organized?
• Thinking about collaboration: what do you think are some of the biggest collaboration challenges within your company?
• What are the consequences of these challenges and how do you solve them?
  

Each of these questions is non-judgmental, open, and historical.

While you may have a knee-jerk inclination to ask more specific questions directly related to your app or service, it’s best to let respondents tell you holistically about their typical day and then infer insights about how your product fits into their lives and existing habits based on their answers.

Note that these questions are devoid of suggestion. Rather than asking how users prepare for meetings right away, our product manager asks, “Do you need to prepare for meetings?”

A subtle but important distinction. He does not assume they need to prepare for meetings at all. While it’s safe to say many people may need to spend time preparing for meetings, this may not be true at certain companies or for certain individuals.

Classify, as a product, is especially useful for people who need to spend a fair amount of time preparing for meetings throughout their day. One of the problems we solve is related to locating key files and information before meetings, spending valuable time preparing for meetings, and experiencing stress or confusion before meetings.

It’s vital that we find out whether a respondent we guessed would be in our target persona actually has the problem we solve for. Making inaccurate assumptions about the prevalence of certain problems among our target persona would lead to an overly optimistic outlook on the size of the market.

The other questions dig deeper into the challenges the respondent faces throughout the day. Our product manager doesn’t suggest challenges, highlight specific challenges, or offer up solutions. Keeping questions open-ended eliminates the power of suggestion from interfering with the results of the interview.

Step 3: Understanding the user’s goals

• What do you consider success in your role?

Again, this question does not impose parameters on how the user should respond. It’s general, open, and neutral.

The respondent’s answer surfaces the results that matter to them in their role and provides insight for our product manager into how our app could potentially help them achieve success more efficiently.

Step 4: Workflow walkthrough

Next, we invite onboarded users to open the app. We walk them through a few hypothetical scenarios to see how they instinctively use the tool.

We also ask questions about what they expect certain tabs, buttons, options, and features to do based on how they’re labeled and designed.

Rather than telling the user how to use the tool, we take note of their assumptions and observe the ways they’re already using it to get a sense for the intuitiveness of our design and determine whether certain features, buttons, or options are misleading or unclear.

It also gives us insight into whether the person we’re interviewing is getting value from the app in the ways we predicted, or if they’re using it for an entirely different use case we hadn’t thought of.

This information either validates our current value prop, disconfirms our previous hypothesis, or introduces new opportunities and use cases for our product team to research further.

Whatever the outcome, it’s all valuable insight for the team that allows for informed product experimentation based on real user behavior.

Three simple rules

In summation, here are three rules from Teresa Torres that we reference when creating templates for user interviews:

1. Avoid speculation
2. Ask for specific stories about the past to generate insights
3. Look for evidence of action to evaluate solutions
   

Ensuring that you’re never asking users about future behavior, always referencing past actions, and observing the actions they take now to evaluate the veracity of your solution is key to getting as many accurate insights as you can from each user interview.

That’s the rundown on structuring a user interview for well-informed product experimentation. Good luck out there.

Oh, and if you’re feeling frazzled and overwhelmed at work, try Classify. We built it specifically for product managers. It’ll help you stress less.