In my last essay I described “nondirected interviewing” and how to write nondirected interview questions. In this essay, I’d like to talk about how to conduct such an interview.
An interview is a funny situation. It’s like a friendly conversation between strangers, but unlike the kind you may have on the bus. When chatting on the bus, people try very hard to agree with each other and to quickly communicate interesting information. Each person wants to be liked and adjusts the way they speak and what they say so as not to offend.
This type of exchange is perfectly fine for maintaining civil society — deeper exchanges can always happen as an acquaintance deepens — but shallow banter isn’t appropriate for an interview. You need to find out what someone is experiencing, what they’re thinking, or what their real opinions are.
In a nondirected interview, the interviewer is more than just a conversation partner; you have to behave differently. You must be more vigilant about the meaning of your words and their implications. An accidental ambiguity can cause you and your interviewee to use the same word very differently; a leading question can guide an interviewee to provide a different answer than what they truly believe.
Here are some guidelines for how to ask questions that will limit (though likely never eliminate) bias:
Define Terms: Words are ambiguous and easily misused. “That thing” can refer to a button, a feature, or the whole site. Personal definitions of words can differ from the dictionary definition or the development team’s definition. Someone may speak of a simple function as a “module,” while the development team may call complex clusters of functions “modules.”
When using a technical term, clearly define it first. Whenever possible, use the respondent’s definition of a word.
Keep Questions Simple: Questions should be clear in both language and intent.
It should be posed to gather raw information, and to uncover assumptions and perceptions. Never, under any circumstances, ask a question to prove a point or to justify your actions.
The goal of each question should be focused, explicit and simple.
Don’t Force Opinions: There are times when people just don’t have an opinion. They may never have thought about a given question in qualitative terms, or they may not have enough information to form an opinion. “Would this be better if it was done automatically?” may not make any sense to someone who has no experience with the process in question.
However, when asked for an opinion, most people will form one on the spot. Such opinions obviously aren’t carefully considered or deeply held.
Make sure that your interview subjects are likely to have an opinion when asking them about your subject matter.
Restate Answers: One of the best techniques to cut through problems with questions is to bounce the respondent’s answer back in different words.
Suppose your interviewee says, “If I entered the name of a teacher here, then they could look at the students in their class, but if I put the name of a principal, then they could see all the scores for all of the kids in their school? I mean, I would have to put something totally different here depending on who it was. I don’t know what to put in here, and I’m stuck.”
You might respond, “So you’re saying that you’d like to know which group the text is referring to because you would expect to be able to do different things based on who it was referring to. Is that correct?”
Rephrasing an interviewee’s responses helps prevent misunderstandings.
Follow Up With Examples: Occasionally, you may want to ask an intentionally broad question to see how people understand a concept or to explore their most general thoughts. People will understand such a question, but may not know how to start answering. Prepare an example (or two) for such questions. After the participants have given their initial answers, you can re-focus their thoughts with an example.
You might ask, “When following up on a news story, what kind of information are you typically looking for?” If the answer is insufficiently precise or shows a misunderstanding, you could follow with, “Can you be a bit more specific? Do you typically look for things like a broader analysis, more facts, different perspectives, or something entirely different?”
Examples give interviewees a chance to answer broad questions with a fresh perspective.
Clarify With Specifics: People have a tendency to generalize from a single experience, or a small number of experiences, to an overall case that may not accurately represent how they behave.
For example, if someone is talking about “shopping carts” in the abstract, ask them about “this shopping cart.”
Using specifics prompts interviewees to break out of biases they’ve formed through similar experiences.
Use Artifacts: Another way to bring interviewees’ focus back to the immediate situation is to ask them about the elements of their environment. Use questions that have to do with the physical objects (or the software objects) they regularly use.
For example, if you notice a lot of Post-It notes stuck to a monitor, ask whether any of them have to do with the task they’re trying to accomplish.
Asking interviewees about their environment discourages them from overgeneralizing.
Understand Your Expectations: It’s impossible to be a blank slate when approaching an interview situation. However, if you’re cognizant of your expectations as you’re asking questions you can avoid leading the interviewee into an answer that he or she wouldn’t have given otherwise.
It’s common for people to say that something was easy or straightforward, when in fact it was quite difficult. They’re just trying to be nice. If you expect that something should be easy, you’re less likely to notice when that’s not the case, and thus less likely to probe statements or actions that imply otherwise.
Be wary of situations that surprise you or when you find yourself predicting the interviewee’s next statement.
Listen: As the interviewer, the interviewees will often ask you questions (especially if they’re using or thinking about an unfamiliar product).
Before answering a question, respond with another question that delves deeper into the interviewee’s experience.
If someone asks, “Is that how it’s supposed to work?” you might reply, “Is that how you think it works?” or “Is that how you expected it to work?”
Asking follow-up questions will help you find the real problem or root of an interviewee’s experience.
Stick With It: Do not fear boredom. Sometimes the questions you’re asking, or the responses you’re getting, seem mundane. You may ask the same question three different ways, and both you and the interviewee know it. Don’t shrink away from repeating what’s important. Interviewees are remarkably understanding, and your goal is not to become friends, but to exchange information.
This doesn’t have to be a chore. With a modicum of practice, the nondirected approach will become a pretty natural way of asking questions.