Semantic Scholar – UX Evaluation

Role: User Researcher
Organization: HCDE, University of Washington
Client: Allen Institute for AI

Project Overview

Semantic Scholar, developed by the Allen Institute for AI, is an academic search engine powered by AI. Our team was tasked with evaluating the Author Influence Matrix—a feature designed to visualize how researchers influence each other’s work. While promising in concept, the tool presented usability challenges around entry points, terminology, and engagement.

Goals

1. Benchmark usability of the Author Influence feature

2. Understand how users engage with influence-based academic tools

Participant Inclusion Criteria

1. Active academic researchers familiar with online academic tools

Research Objectives

1. Understand how users navigate to and engage with the Author Influence Page.

2. Identify pain points and opportunities for improving comprehension of influence metrics.

3. Assess the value and usability of sorting/filtering features (e.g., acceleration, velocity).

4. Explore the potential for the page to support broader discovery and research insights.

5. Gauge trust in the system-generated author connections.

Methods

We applied a mixed-methods approach combining exploratory qualitative research and summative usability evaluation.

Key Methods Included

1. Contextual user interviews

2. Task-based usability testing

3. System Usability Scale (SUS) survey

4. Thematic analysis of qualitative data

Participants completed a series of tasks, such as navigating to the Author Page, interpreting influence graphs, and using filters. Post-task reflections and the SUS survey provided additional insight into satisfaction and perceived usability.

Key Findings

Navigation & Wayfinding

1. Users had difficulty locating the Author Influence Page from search results.

2. Labeling and entry points were not intuitive, particularly for new users.

Comprehension of Influence Metrics

1. While users understood the concept of influence, many found terms like "velocity" and "acceleration" confusing without additional context.

2. Visualizations were engaging but lacked explanatory affordances.

Exploratory Use & Discovery

1. Some users stumbled upon unexpected and valuable connections between authors, indicating potential for knowledge discovery.

2. Others struggled to understand how to use the feature to guide their research.

Filter & Sorting Tools

1. Filter tools were underutilized due to low visibility and unclear functionality.

2. Users desired more control over filtering by citation type, year, and publication quality.

Result

Redesign navigation flows to clearly surface the Author Page from search and author results.

1. Enhance labeling and provide on-page tooltips or onboarding elements to explain influence metrics.

2. Improve filter design by prioritizing commonly used research attributes and making them more visible.

3. Reframe visualizations to encourage discovery, possibly through thematic clustering or keyword tagging.