Observing Observation

By Lauren Allen, Learning Media Design Center @CMU

Cognitive Task Analysis & Contextual Inquiry

Unknown specimens in their vials. Some taxonomists knew family and had a good guess on species without even removing specimens from the vial. Others worked through the dichotomous key to find family and genus classifications.

Cognitive Task Analysis (CTA) and Contextual Inquiry (CI) are two different research methods used to examine the practices of experts as they perform specialized and complex tasks.

​
CTA comes from the world of cognitive psychology and learning sciences, where researchers seek to understand the specific moves and decisions that experts make, explicitly and implicitly, as they use knowledge, skills, and background to complete their task.
Contextual Inquiry is a HCI/design user research method, and focuses on expert practices and their workspace, and seeks to understand how, when and why they use particular tools, resources, and acquired skills to physically perform their task. 

For this project, we combined data collection for CTA and CI (which we've dubbed the CTA/CI study) on eight different freshwater macroinvertebrate taxonomists with varying levels of experience in taxonomic classification and training. These taxonomists were identified through what qualitative researchers sometimes call "snowball" sampling--we asked our contacts in the field of taxonomic classifications (i.e. our collaborators on this project) if they'd be willing to participate, and if they could recommend someone who had experience as a taxonomist and/or trainer of taxonomic ID whose process would also be useful for us to understand. 

For each subject in this study, we observed them working in their "natural environment" AKA the context in which they usually do their taxonomic classification work. This always was a space with a microscope, and at least one dichotomous key (if there was only one, it was always Merritt, Cummins & Berg's Aquatic Insects of North America). We set up our audio and video recorders, and I asked a few introductory questions about their background in entomology and taxonomy, and what they found to be difficult and motivating about the work. Then, I presented them with a series of three unknown preserved specimens in order from easiest to hardest (based on the judgement of the person who provided the specimens, one of our entomology partners on the project).

---

​If they asked, I told them where and when the specimens were collected (not everybody asked). As they proceeded to identify the specimens, I asked them how and why they were making particular decisions. After three specimens (or an hour and 15 minutes had passed), we moved on to a computer simulation of the same task, with a high-resolution image of an unknown specimen set to a very zoomed in level (so only part of the specimen was visible at a given moment). This simulation was intended to force experts to use the mouse to move the specimen around so we could more easily see what parts of it they were looking at to make their identifications. 

Simulation Specimen 4, available view. Here the taxonomist can see one eye and part of the thorax of the specimen, but not the entire specimen at once.

The purpose of CTA is to dig into the why questions of how experts complete complex tasks--Why do they decide to choose one side of a couplet in a dichotomous key over the other--is it because of something they observed visually? Or because of something that they know about the range or habitat of a specimen they're identifying? Or is it because they know something else about the specimen that will come into question in a couplet further down the line? Some of the experts in our study barely touched their dichotomous keys when we asked them to identify unknown specimens of freshwater macroinvertebrates down to the genus level. Some of them knew what family a specimen was part of before they had looked at it under the microscope. Others were able to identify a specimen's order, and make an educated guess about it's family, and then would use a key (or, in some cases, several keys) to complete the task of taxonomic identification.

Contextual Inquiry's purpose is to get at the how questions of expert practice: How do experts use tools like microscopes, forceps, and fingers to get the information they need about a specimen? How do they refer to books, dichotomous keys, publications, and websites to confirm or disconfirm their guesses about a specimen's family or genus classification? How do they interact with their colleagues in these efforts? It's a much less "invasive" research process--the design researchers on our project (Marti and Ahmed) asked only a few questions, but took lots of photographs and observational notes when we watched experts at work. 

CTA is an in-depth process of questioning and probing an expert as they engage in (or reflect on) a task or problem to solve. It is well established in the cognitive science research world that experts are not easily able to articulate every step of their cognitive processes as they perform complex tasks in their area of expertise--part of being an expert is having practiced enough for some aspects of the task to become "automated" and therefore not something that the expert is aware of doing. Sometimes automated aspects of practice are muscle-memory skills, and sometimes they're simply looking at something and knowing what it is without having to analyze it's component parts. CTA methods allow the researcher (in this case, me) to ask questions, or probes, of experts as they are performing the task that provokes the expert to verbalize why they are doing certain things or what background knowledge they are accessing to take the next step in their process. During the process of this study, the most common thing that I said was "What do you see on that specimen, or what knowledge are you using to make that decision?" ("that decision" being the taxonomic identification or the decision to take one side of a couplet when using a dichotomous key). ​