Research Methods and Analysis
Sampling: Anthropology vs. UX Research
The first thing I was taught related to research methods was sampling. I learned all about probabilistic and non-probabilistic samples. I heard stories about snowball sampling and the ethical implications of Humphreys’ Tearoom Trade sampling techniques. Mostly I learned that anthropological studies often relied on small samples to uncover rituals, tokens, beliefs, and struggles of a given population.
Qualitative User Experience Research also relies on small samples to uncover rituals (and workflows), beliefs (and motivations), and struggles (spoken and unspoken) of a given user base. I look for patterns in small sample sizes to provide inspiration for product changes and innovations. How I do that, again, is not that different from how any anthropologists would approach a research project.
Methods within the Disciplines
I was taught a number of methods in my anthropological training. Structured, semi-structured, and unstructured interviews are used to explore individual’s self-reported understandings and behaviors. I learned how to carry out focus groups (and when not to), card sorts for organizing information, and surveys. But the most powerful method in our toolbox is observation – participant or otherwise. Observation allows the researcher to compare what is said and the actual behavior.
Qualitative User Experience Researchers utilize 1:1 semi-structured interviews quite often. One might consider it a way to explore the individual experiences of which Weber was so fond. However, observation still rules in the UER world. Whether it be observing someone using a website to evaluate usability or watching folks getting ready for their day, UERs strive to provide Geertz’s thick description for their company and stakeholders as inspiration for product changes and innovations.
Approaches to Analysis
Analyzing qualitative data relies on tagging instances in data in a consistent way so that the researcher can find themes, find quotes, find the story. After spending six months in the field “hanging out” with small fishing [community?] town members, the anthropologist will transcribe all of her observations, interviews, and field notes. The next step is to read through the transcripts at least three times, adding comments and tags. Eventually, the story of the fishing [town] culture emerges, and the anthropologist writes articles and likely a book. This is, of course, quite a simplification of what actually happens during the analysis stage, but it captures the major milestones.
A Qualitative User Experience Researcher follows a similar pattern. After observing or interviewing 5-12 participants, a UER will transcribe what she has seen and heard (or use the video as a transcript). Then comes the reading/watching several times over until the story emerges. One rather large difference between anthropologists and UERs can exist because UERs oftentimes engage in some sort of co-analysis with their stakeholders that involves transferring data points into sticky notes and moving them around into themes clusters. Even when this co-analysis happens it still falls to the researcher to interpret the themes and to tell the story that emerged.
Which Method When
Academic anthropologists often secure funding for their research and investigate research questions of their own making. Applied anthropologists, on the other hand, are often sought out to complete research related to a specific problem. This often means coming into a project sometime after its inception and being asked to either interpret what has been found or design a study in line with the established goals of the organization.
During my time in graduate school, I was a researcher for a number of nonprofits, helping them to understand things like how to make a community farm successful at selling their produce, the impact of community gardens, and the motivations of donors participating in charity fitness events. Some of these projects were scoped and designed before I joined the team, others were meant to save a sinking ship, and at least one gave me the opportunity to participate early and often. Nevertheless, it is almost always the research question(s) that guides the choice of method. That is, there is not an overarching process that the researcher must first consider before turning to the research question(s). In this way, the choosing of the methodologies differs between traditional applied anthropology and user experience research.
Qualitative UER maps to a design process, and while the chosen methodology is guided by the research question(s), it is first anchored by where you are in that process. For those unfamiliar, a typical design process is what gets labeled the “double diamond.” You have an idea or something you think is a problem in the world so you diverge and try to understand, “Is it a problem in the world? Do people care that it’s a problem? Are there similar problems in different areas of life?” You then start to converge on the actual problem. Once you are certain of the problem, you once again diverge and generate as many solutions as you can possibly imagine. Through intuition and testing you can begin to converge on a viable and desirable solution. Viola! You designed a product or service that people actually want (in theory).
A qualitative UER must first know where in the process their stakeholders are because that affects the approach and methodology. For example, if all they have is an idea of a problem, a UER would want to understand if that problem truly exists. So one method to use could be observation and interview. If, however, they knew that a problem existed and knew the information it took to solve it but not much else, I might want to do a card sort to help them arrange the information in a way that makes sense to their users. There are plenty of resources online about this. Here’s one from Nielsen Norman group.
A stakeholder is anyone who has a vested interest in the experience you are researching. In school, my stakeholders were my professors and whatever community organization I was working with at the time. The process was quite simple:
- Meet with stakeholders to discuss what they wanted to understand better.
- Design a research plan to answer the question(s).
- Execute the research plan while keeping my stakeholders informed.
- Conduct the analysis.
- Share out the results to all stakeholders
What was often missing that is crucial for the embedded research (i.e. researcher working within a corporation or company) was step six – ensuring your research had some sort of impact. When I was in graduate school I often handed over my research findings and hoped something would come of them. However, part of what you are evaluated on inside a company is your ability to generate results
In the tech industry, “stakeholders” usually mean designers, product managers (or Product Owners), engineers (or developers), and marketing (sometimes called product or digital marketing managers). The addition of the ‘ensuring impact’ step actually affects the rest of the process because as an embedded UER, I have to be thinking about impact from the very beginning. Just imagine a group of people spending every working minute thinking about a given product. Now you – a UER – goes away to study their users and comes back seemingly out of nowhere to tell them what they had done wrong and how they might do it better. That would not go well. Rather imagine you invited that group of people to participate in the forming of research questions and materials, had them join you in the lab or field, and then asked them to help you make sense of the data. Then telling them what might be wrong is more of a formality because they saw it themselves, and with their minds open you can then suggest frameworks and principles to guide future thinking.
I hope this short comparison has helped illuminate how the skills of applied anthropologists transfer to qualitative user experience research with almost no skill gap. Importantly, there is a field of quantitative UER, which is not as good of a fit for most anthropologists, so read the job description first! Before I finish this piece I would like to describe a day in my life for those curious about UER.
A Day in the Life of a UER
Learn Outlook and Gmail. Use the calendar features. I live and die by my calendar, meaning if it’s not on there it likely won’t get done. I would also recommend utilizing Trello. With that said, let’s go.
I start work by checking my email. Anything important either gets responded to right away or I put a task in Trello under my “Today” column to reply before EOD (end of day). Once my inbox is tidy, I likely have a meeting to attend. It’s usually a meeting with some stakeholders to discuss potential research, a fellow researcher to discuss some collaboration, or a general meeting to update the team on some product I have recently researched.
After the meeting, I check my email again. With that in order, I check my Trello tasks. These tasks are very likely steps in the research process: schedule kickoff meeting, create interview guide, schedule participants, analysis, create deck (PowerPoint, Keynote, Google Slides), schedule share out. So I work on the most important things until my next meeting or lunch, whichever comes first. After lunch, I likely have another meeting to discuss some aspect of research (I’m working on a number initiatives because of my”senior” role). In the final hour or so of the day, I check my Trello and decide what must get done today and do it. Then I go home.
This obviously looks different when I am conducting research. I am actually writing this post on my phone “in the field” while I lead a week’s worth of customer visits in Seattle. I also didn’t mention all the hallway conversations, random discussions about movies and pop culture with my those who sit around me, or the design-wide meetings where we learn what other designers and researchers are working on at the moment.
Nevertheless, my day in its most basic form is meetings, planning, email, and more meetings. What does your day look like? What did your formal or informal training look like? Leave us a comment and let us know. Thanks!