Using Log Visualization to Interpret Online Interactions During Self-Quizzing Practice for Taxonomic Identification by Chelsea Cui, Jordan Hill, Marti Louw, Jessica Roberts.
We were excited to present CMU alumna, and former REU student Chelsea Cui’s study at the 2021 annual meeting of the American Educational Researchers Association (AERA). Chelsea analyzed log data from participants in a 10-day study in which they used our quiz feature to practice aquatic macroinvertebrate ID. We analyzed interactions with the quiz tool along with accuracy in pre- and post-tests to determine how the quizzing platform was being utilized by learners at different experience levels. Through the custom visualization platform we were able to glean insights on how to improve our quizzing platform for future use.
See the poster here: https://aera21-aera.ipostersessions.com/Default.aspx?s=D0-9D-A0-1B-C0-49-87-5B-8A-3F-56-A7-02-C5-ED-66#
By Dominique Aruede, CMU Cognitive Psychology
When I joined, the quiz section had not been developed at all in favor of refining and refreshing the ID key and field guide which are more directly interpreted from the original website. Consequently, it was important to gather human-centered data again in order to assess the efficiency of the current design and determine if we could branch out to the quiz or keep working on the current designs.
First Task: Audubon Field Guide Walkthrough & Modeling
I spent a little time doing some exploration with a physical field guide and noting how it maps to real-world use and facilitation of citizen science. This step was particularly useful for getting my bearings on what the purpose of the Macroinvertabrates.com project is, how insect identification fits in, and how it works.
Second Task: User Testing
We conducted several user tests on insect ID experts and novices alike, with a focus on novice user experience from high schoolers and other young students.
The purpose of these usability tests was to gauge the design direction of our second iteration and to gather more user input to launch into the next iteration.
We reused the protocol from the first round of user testing and changed the questions to capture answers on our new inquiries:
The results revealed to us that the field guide was indeed informative, but subsequently neglected some utilitarian features that would be helpful for novice users. This helped to inform the learning goals and first iterations of the quiz design.
Below is an affinity diagram of the synthesized insights. It includes
Third Task: Quiz Design
The earliest version of the quiz was a rough sketch I drew up in Figma, but it did not have any learning goal or research claim behind it besides identifying some image as belonging to an order. These are the sketches below.
After further visualization, we wrote out the learning goals we thought would be the best to target in quiz mode based on the insights gleaned from the previous affinity diagram. The learning goals are summarized here:
After we gathered our quiz references, including Quizlet and Duolingo, we used our learning goals to consolidate designs for one flow of the quiz section. Users select their quiz type first, then their learning goal. An alternative flow that we are yet to explore involves switching those two options. Then we incorporated three quiz types: a flash card review, multiple choice quiz, and a matching quiz.
We drew this up on Figma and discovered a few issues/limitations
Afterwards we consulted our collaborator, who is an expert in aquatic macroinvertebrate education to give her opinion on the content and direction. From her insights, we decided to scrap learning PTV as a learning goal due to the inaccuracies of generalizing at the order level.
The next steps we plan to take include developing all the content necessary for accurate quizzing, fleshing out the flashcard review, the between order quiz learning goal for all quiz types, the common features learning goal for all quiz types, the common mistakes learning goal for all quiz types, exploring a new learning goal: “learning life history and fun facts about insect orders” as a deck in the flashcard review, and testing the usability and flow of our current design.
By Dominique Aruede
Last week Dr. Louw tasked the design team with exploring the material design of the Audobon Society Field Guide and Pocket Guide and possibly perform a few insect identifications using the guides. The latter book was a smaller, simplified, more utilitarian version of the pocket guide.
In an effort to frame and understand the flow of a typical bug ID guide, we spent a week looking into the specific design choices of Audubon identification pocket guides and field guides. Alice, Estelle, and I each went though the Field Guide, noting down different features that added value or clarity or ease of use. We all agreed the field guide had general helpful features like annotated images and a glossary of all the common names.
Since I'm a very new member to the team I received another book, the pocket guide, which seemed to be for more novice audiences and was easy to digest for context purposes. The field guide was more advanced in technical information than the pocket guide, and that took away some of it's practicality. Below are our individual syntheses and visualizations of the insights produced from the objective.
By Estelle Jiang and Alice Fang
Moving to a higher fidelity prototype by following the design system.
After showing our low-fidelity prototype to the entire team and developers, we decided to move forward by applying a more detailed design and developing the visual + design system. We also figured out how to showcase the relationship between orders, family and genius on the mobile application, and the UI components for each type of 'page'. It was one of the biggest challenges we met previously.
For the color theme, we followed the guidelines the project used on the website and applied the blue color to highlight actionable parts. To keep the app clean and concise, we used white for the major user interface design. As for the typography, the body font is Nunito Sans and the title font is Roboto Slab. Due to limitations with the database information, and in an effort to bring about the features of the specimen in photography, we also worked with the gigapan background color, creating a floating, borderless 'under a microscope' look [see image on the right].
Changes we made for the high fidelity prototype after discussion:
Homepage - We thought the card view can be bigger to attract people’s attention and intrigue their interest. Since we only have 10 orders, we did not have too many concerns about accessibility at the very beginning. The ID key button is also replaced on the home page.
The dropdown menu was also changed to help user easily navigate and get back to main page.
We also added icons to explain the functionalities and applied color for the side menu to make it stand out more. As we mentioned on the previous post, we were struggling between a button to expand, and swiping up. Since we were worried about the experience of swiping up which is too hidden on the bottom of the screen. we iterated and created a button on the bottom for accessibility instead.
Planning and preparing first round usability testing
To conduct our first round of testing, we started with writing the testing protocol, thinking about the purpose of the testing and the goals we want to achieve.
The purpose of our testing was to test the logic of the user flows, and to identify potential navigation and usability issues. We wanted to understand if the application is engaging to users, and is useful in identifying macroinvertebrates and learning their characteristics.
We assigned a few small tasks for users to finish during the testing:
Pre-task: Users will be given 30 seconds to get familiar with the application before doing task.
First task: Users are asked to browse the different orders though different ways. This way, we can then tell whether the design makes sense, and take note of how users navigate through the different levels of information.
Second task: This task was focused on the detailed Order & Family page designs, users are asked to find out more detailed order and family information, as well as specific diagnostic characters for a specific family. By asking the users how difficult the task is, we can evaluate the slider design idea we had, and how accessible / noticeable the actionable button is.
We also asked additional questions at the end of testing to check whether they can have a clear understanding about our application throughout the testing process, including:
Synthesize the findings to guide our next iterations.
Overall, we conducted five user interviews, with macroinvertebrate experts, people familiar with the site, and a novice user; I also got feedback from my friend who knew nothing about the concept or field in order to get additional novice learner’s insights.
Rather than use the normal user research method - affinity diagramming to synthesize the testing findings, Alice made the excel sheet to list out the key points the interviewee made for each task. It helped us highlight the common suggestions and feedback.
Here are the findings that guided our next iterations:
Iterations we made.
Since the first version of high fidelity prototypes are hard for novice learners to learn and understand the additional information, I quickly brainstormed two other versions to display the information and hierarchy between orders and families.
The first version allows users to swipe and learn along the way. The experience is more immersive and easy to follow if the users have no idea about the insect and the order. However, it doesn’t give enough freedom and choices for users to explore themselves, and quickly becomes repetitive for more experienced users.
The second version can cater the needs of both experienced users and novice learners since it allows them to quickly switch between various levels of information. The structure of the insects are also easy to tell and discover. The homepage was also iterated from showing only one order to display multiple orders at once.
The developers moved forward with this version, and worked to develop a beta version. It was interesting and difficult figuring out how to work in parallel; they were focused on setting up the database and structure, while we were iterating through the designs, but we couldn't progress too far or change too many things after they began developing the pages.
By Alice Fang
New 🐞 bandanas! I designed a bandana that highlights the insect illustrations done by Morgan Summerlin, inverted on black. The illustrations are placed on a diagonal for optimal bandana-folding-wearing! :-). Handy for working in wet settings, and promote Macroinvertebrates.org when you have to maskup,
By Alice Fang
Re-framing with a new team!
Chelsea and I were joined by Estelle (Yi Cheng) Jiang and Dakota (Zi Qi) Dong in late May. Moving forward, Chelsea and Dakota will be working on development, while Estelle and I will be working on the design and user-testing of the app. There was an adjustment period as we introduced the project to the new members and got everyone acquainted while figuring out how to work collaboratively in a remote wfh environment.
Through this process, we've been utilizing google docs and spreadsheets, as well as Figma, but coordinating between design and development has been tricky. Accommodating and synching the design and development timelines was difficult, and it was a bit touch-and-go.
Spreadsheets and Organizing Data
Previously, I struggled with establishing a structure to the app that allowed for navigation in and out of orders/families. To make the taxonomy clear (as we are non-scientists and non-bug experts), and to organize the information for design purposes, I created a spreadsheet with the following:
1. Inventory of functions that exist on the website
In order to figure out the minimum viable product that can be developed by students within a summer, and to compare what needs to change from Order to Family, I listed all of the functions for Order/Family/Genus.
2. List of insect orders, the families in each order, and the number of specimen (genera) in each family, and the number of diagnostic characters for each family and order
While many of the families have similar numbers of specimen, there were a lot of outlier cases we needed to keep an eye on, and account for while designing. The length of names, both common and scientific, impacts the typographic system, and the number of diagnostic characters affects how we visually set up that information. Some of the cases are as follows:
*Dev team has to set up the database (I don’t know the exact details). We ran into a challenge late in the summer [early August] where the Gigapan database had to be moved, and there was no way to extract some of the family traits and text information, requiring manual copy and pasting
Collecting References and Resources for Ideation Round II
Getting the design team on the same page.
I showed Estelle the previous mockups and ideas that I had for the mobile app, but in order to refresh and sort of create a mutual visual language, we took the time to research and look at other apps. We compared and discussed field guide apps, quiz apps, and other text-based apps like news/media.
*Referencing Headspace, AirBnB, Medium, WWF Together
*Referencing Twitter, Google Maps, Medium Merlin ID, Audubon
Lo-fi Version 2
New Ideas and Changes
Estelle quickly mocked some basic page structures, and documented user stories. This allowed us to see possible entry points and user profiles for the app. What are possible ways people would use the app? What would they be looking for? And how do we prioritize that development at the same time?
She also created this flow diagram of entry in the app and access points to different functions; however, it didn't include navigation and returning to previous pages, or other major functions we hope to implement in the future, like a quiz.
*Comparing what a bottom pop-up that scrolls up, and a button on the bottom, would look like.
Between a button to expand, and swiping up, we originally decided on a swiping up action, but were worried about the screen experience that it would be too hidden on the bottom of the screen. As we moved into user testing, we created a button on the bottom for accessibility instead. It stood out more, and for someone holding a phone, was located in a position that was easy to access.
Through this process, we also started to get into the look and feel of the application. As the focus and beauty of the collection is the high definition images and close-up thumbnails, I really wanted typography to play a role in the visual style while staying close to what the desktop site looks like.
by Alice Fang
A little after spring break, Chelsea and I started on conceptualizing and developing a mobile app version of Macroinvertebrates.org, which we will continue to work on into the summer.
The original question we started with was what should the app do? (and what can we do to create a minimum viable product as a 2 person group?) The primary functions of the site that could be reflected in a mobile app were:
Primary Goals for mobile: learning orders! and providing access when you don't have internet connectivity (creating a lighter weight, downloadable version)
A mobile version of the ID Key was already prototyped by Jaclyn last year, so the majority of the focus will be working on integrating Order and Family level information. Based on how far concept/prototyping goes, we may also integrate a practice quiz as well.
The first issue we ran into while refining the ID Key was the number of cases that the original design couldn’t accommodate for. For example, in the ID Key on the site, there were a few questions that examined/compared two different physiological traits ("wings OR wing pads", instead of just "wings"). How could the gallery of characteristics show images of both, and how should the help button appear to show both text explanations as well?
This was the first (and definitely not the last) time where I struggled designing for mobile, instead of desktop or 'website on mobile.' I don't have much experience in designing a purely mobile app product, so I'm excited for the challenge and learning experience.
Areas that need resolving as we move forward!:
Looking at Mobile Interactions
I took note of some common features of mobile apps to help structure our app as well. These are some references that could help with structure/organization.
Organization + Hierarchy
Once a user selects an order, the organization from Order > Family > Zoomable Image (Genus) becomes a little confusing. Should the user be able to select into a family page (option 1) or does it lead directly to the zoomable image? At what level is family-specific information displayed then?
*We established early on that Genus-level information was probably too specific for mobile, trying to keep the app more general
By Emily Chan, Summer 2019 NSF REU
One of my main projects this summer was designing a practice game that leveraged our large collection of high resolution images for volunteers to practice identifying macroinvertebrates.
A tool for practicing macroinvertebrate ID has been one of the most requested features by volunteers and educators alike. Before getting started developing a game, we took a look at the current resources available online. While there are an abundance of identification-based practice games on the internet (though hardly any for macroinvertebrates specifically), many of these games have similar designs that use low resolution images or generic illustrations, have little possibility for user visual interaction, and use a handful of example specimens that never change, meaning users can usually only play the game once or twice before being able to memorize the answers. In addition, usually only four choices of answers are provided, multiple choice style. Examples below:
In contrast, when conducting ID activities in authentic citizen science practice, volunteers need to be able to identify a large number of macroinvertebrates that may be sampled from a stream, to work around variations in size, shape, and color, and to be able to choose the right identifier from a large list of choices.
Our goal in creating this game was to simulate authentic practice, so our design for this practice game was informed by the identification process for water quality monitoring, and the biotic indices used by volunteers to classify macroinvertebrates.
One of my goals was to keep the game as similar to the actual website as possible, so that it’d be easy for Chris to actually implement. Therefore, my first prototype mimicked the panel design developed by Alice. Every mystery specimen would be a genus-level page, with the text in the panel replaced by a question panel. The key ideas in the paper prototype were: the game goes on indefinitely until you choose to stop (in order to take advantage of our large collection); after you finish, it compares the water quality rating you would have calculated based on your identifications with the actual water quality rating; it uses the diagnostic characters as hints; when you get an answer wrong, it provides information about the differences between the answer you chose and the correct answer; and it has all the same zooming and changing view functionality as the regular site.
Though every single practice game I looked at provided four choices you could choose from, it seemed that this might not make sense for our game--if the choices were randomly selected, we might end up in a situation where the user is asked to identify a mayfly, and the choices are clam, leech, beetle, and mayfly. Having choices that don't actually look similar to the specimen isn't particularly helpful for the user's learning, but if we wanted to only show visually similar choices, someone would have to manually create the choices available for each specimen, which seemed excessive. In addition, in the actual scenario, volunteers need to choose the right identifier from a very large number of choices. Therefore, I sketched a multiple choice version as well as a version with a dropdown menu.
XD PROTOTYPE 1
We collected feedback on this first XD prototype at the Macroinvertebrates.org site launch event, allowing educators and trainers to play around with the mockup and asking them about their thoughts. Their main concern was whether the game would be leveled--suggestions included student, volunteer, and expert levels. Who the target audience is turned out to be a central question that the game still struggles to fully address. We chose to focus on volunteers, and get a working version of the game for that audience first. I also considered whether it'd be useful to separate by region or ecosystem (like the Cornell Bird Quiz does), but I didn't have enough information about who'd be using the game and what they'd want from it to make a judgement about that. We made some additional minor changes to the design based on suggestions, such as bringing the diagnostic characters back on the correct/incorrect page, including an illustration of the wrong answer as well (i.e. a small illustration of stoneflies next to "This is not a stonefly"), and trying alternatives to "incorrect" (e.g. "try again", "sorry", etc.).
XD PROTOTYPE 2
After getting feedback on the first XD prototype at the Macroinvertebrates.org site launch event, we made the following changes:
XD PROTOTYPE 4
For the next prototype, we decided to make use of the annotated illustrations that we had available (with the text enlarged) to replace or complement the informational text.
XD PROTOTYPE 5
After consulting with Chris, our developer, we shifted towards a tab-panel design, to maintain consistency with the genus level pages and to make developing the game easier. The hints were shifted to the second tab, similar to the genus-level page, and the third tab would either be an "About this Game" tab, or a score-keeping tab.
Chris also informed us that the instructions and results would have to be shown on a splash screen. Below are examples of splash screens we considered:
XD PROTOTYPE 7
After getting more feedback from Tara, we decided to add a list of the ones you'd gotten incorrect in the third tab, with pictures of the specimen and the correct identifier. She also advised us to shift away from having the game be a simulated stream dump (i.e. calculating a water quality rating at the end), and rather focus simply on identification, since there were so many different systems and standards used to measure water quality.
Examples of different icons we considered for the tabs:
Minor changes for the final prototype included simplified icons and adjustment of the correct/incorrect screen.
FEEDBACK + RECOMMENDATIONS FOR FUTURE CHANGES
As it stands, the current game live on the site still needs to have the list and images of the incorrect specimens added, as well as small changes such as adjusting the category names (i.e. "Net-Spinning Caddisfly Larva" to "Caddisfly Larva (Net Spinning) "), swapping in illustrations with larger text, etc. Another major challenge is finding a more elegant way of dealing with the "None of the Above" category. User said that they enjoyed the challenge of identifying uncommon specimens, so I don't think that omitting specimens that would fall into that category is the solution. Since most of the "None of the Above" specimens were either true bugs or fly larva, one user suggested a "True Bugs" category. If future changes are to be made, I would recommend adding a "True Bugs" and a "Fly Larva (Other)" category. Other user suggestions included adding scientific names, since many who used the game were more familiar with scientific names over common names.
However, the majority of user feedback centered around the lack of levelling in the practice game. Since the users were recruited from a broad base of experience levels, many wished that there was an expert version for identifing to family or genus; while others noted that they hadn’t been taught the distinction between particular taxa levels of macroinvertebrates, and felt such categories could be confusing. Future work includes producing a “student” (order-level) and an “expert" (genus) level game.
Users praised the custom-written informational text and illustrations on the correct/incorrect screen as well as the zoom and hint functionality. In our sample, no users expressed issues with the tab structure, start screen instructions, or the indefinite nature of the game, which were design choices we sought to validate.
EXAMPLE EXPERT-LEVEL PROTOTYPE
Hi! I'm Alice, a rising junior at Carnegie Mellon University studying communication design with a minor in professional writing. I joined the macroinvertebrates team this summer as a design/research assistant, and I’ve been working on some site iterations for macroinvertebrates.org the past three weeks—namely, building stronger relationships between icon navigation and the actual informational content. I’ve learned a lot about prototyping quickly and keeping my work organized, in order to present solutions that have to rapidly be pushed to development. Having learned some basic web design and HTML/CSS, it was also a good learning opportunity in working with a developer to figure out what can or should be coded.
On the previous version of the site, navigational icons floated above the panels that held important and interesting content. We wanted to change the structure for organizing the content without creating a major overhaul of the previous system.
Some of the questions I had to figure out for creating a side panel:
One major issue was having a system where all of the content (order/family overview, media, iNaturalist photos, life history, characteristics) could be visible together but still consolidated. The original site had some strange floating/scrolling rules in order to fit all of the information. To address this, I explored different ‘drawer’, or side-panel, designs.
In the end, we took the ‘Media’ content and the iNaturalist photos and grouped them into their own separate category, which was then referred to as the Media Tab. The panel thus had three tabs at the top: Information, Diagnostic Characteristics, and Media.
In trying to incorporate the ‘clear all’ function, to show the macroinvertebrates without any identifying information, we took advantage of the structure of the panel; when the panel closed, it also cleared all information, leaving just the images of the macroinvertebrates. The default then became having the panel open to the diagnostic characteristics tab.
I also played around with different hover cues for the diagnostic characteristics. Clicking on a characteristic opened up a gallery that compared that characteristic for the selected macroinvertebrate with others that had the same feature.
By Emily Chan, Harvard University, Summer 2019 REU
Hi there! I'm Emily, a rising junior at Harvard studying computer science with a minor in statistics. I'm a week and a half into the HCII REU, and I'm working on two projects with Marti Louw, one of which is Macroinvertebrates.org. I came to the HCII REU because as a computer science major and an artist, I'm interested in learning more about design, and I also want to experience what doing research is like.
Some of my personal goals for the summer are to understand how and why design choices are made, get hands on experience developing tools for users using design principles and user feedback, make something that I can show people/people will actually use, and learn about prototyping with smart devices.
My tasks for Macroinvertebrates.org for the foreseeable future include doing usability testing for the new dichotomous key, glossary, and search functionality; creating an improved popup kit for museums to use to engage kids in macroinvertebrate identification, trying to leverage the high quality media that we have and potentially using smart technologies such as NFC tags, etc.; and possibly working on developing identification quiz practice functionality for the site.
To gain an understanding of the design challenges at hand, we did an identification task using the current resources available to volunteers--field guides and key printouts. The sheer amount of materials was slightly overwhelming, so as someone with no experience doing macroinvertebrate identification, I just chose two field guides that had pictures and stayed with those. I identified five bugs, four using the field guides and one using the website. What I noticed when using the field guides was that the detailed text descriptions of the bugs was essential for family level identification, but the volume of the text also gets in the way when it comes to order level identification. My process was to first do visual pattern-matching to the order level, and then read the descriptions of ones that looked similar to the specimen to find traits that would help distinguish between families. I was surprised by how identification wasn't as difficult or time-consuming as I anticipated, but I also didn't use most of the materials, and never used a key.
When I switched to using the website, I tried a similar approach, but the website's organization was slightly different from a field guide (I couldn't just flip through quickly to scan for pictures and then selectively read descriptions), which took some adjusting to. The home page worked well for facilitating my initial pattern-matching approach, but I can see how if I were trying to identify a ambiguous looking larvae, I would want the home page to be more explorable (to be able to zoom in more), since some of the pictures were quite small. Once I clicked through to the order page, I definitely don't think I would have known how to get rid of the diagnostic characteristic boxes obstructing the actual pictures that I wanted to look at. However, since I was already familiar with how to do that, I was able to match the specimen to the common burrowing mayfly just by appearance, but I wasn't able to find the characteristics that might help distinguish between similar looking families in a convenient place.
For me, this raised the question of whether the purpose of Macroinvertebrates.org is primarily to be a learning resource or an identification resource.
An interdisciplinary team