Gutenberg Technologies
TESTING THE DISCOVERABILITY & USABILITY OF GT AI TOOLS WITH EYE-TRACKING
Gutenberg Technologies (GT) is a company that offers a content management platform designed for creating and distributing educational and professional content. Their platform is used by publishers and corporations to efficiently produce learning materials for both print and digital formats from a single workflow. Using AI-powered solutions, GT helps transform static documents, like PDFs, into interactive, multi-format courses. Our client’s goal is to gain an understanding of how people perceive the AI edit tool and also evaluate the “Generate questions and learning activities.”
-
This was a user research project focused on the usability and discoverability of the product’s AI content editing and content generation tools.
-
Gutenberg Technologies
-
The team was comprised of 4 User Researchers, all graduate students of Pratt Institute.
-
UX Researcher
-
Client briefing and goal alignment
Current AI tools analysis and testing
Recruiting test users
8 Eye-tracking studies with Retrospective Think-Alouds.
Analyzing test results and consolidating findings
Designing and prototyping improvisations and recommendations
OBJECTIVEs
Our primary goal was to evaluate the usability and overall user experience of the AI Content Management System (CMS), with a specific focus on the "Edit with AI" feature. We aim to ensure the platform and its new AI capabilities are intuitive, accessible, and powerful for all users, regardless of their prior experience with AI.
PROJECT JOURNEY
user group
For this study, eight participants were recruited that fell within one or more of the following brackets:
HYPOTHESIS
research methods
research OVERVIEW
USER TASKS
FINDINGS & recommendations
The AI tools suffer from low discoverability and a lack of systemic clarity, negatively impacting the overall experience of the platform.
FEATURE DISCOVERABILITY
FEATURE DISCOVERABILITY
While everyone was able to find the “Edit with AI” button, it took a long time, indicating low discoverability.
Participant 1 struggling to edit the quiz. She clicked through several different options to find the one she needed.
Streamline the editing flow by consolidating the editing tools into one menu and adjusting
its position and icon to align with our users'
mental models.
CTA for “Generate with AI” is not easily discoverable.
Combine the Generate with AI options with the “+” section on the right toolbar.
FEATURE USABILITY
FEATURE USABILITY
Participants had difficulty distinguishing the AI generated content and the original content.
Participant 8 was trying to read through all the content to identify differences.
Side-by-side labeled versions of original text and AI-generated version with color coded identification of what was changed or added.
While most participants successfully used the AI features, unclear AI terminology led to frustrations when completing tasks.
Simplify AI jargon, clarify technical options, and enhance action clarity with visual context.
supplemental insight
supplemental insight
Although task completion rates were high, some participants felt the system's "Save" rules were confusing.
Participant 2 trying to find the “Save” button
Notify users of the system feedback and logic with an informational popup.
Overall, these insights show that Gutenberg’s AI tools are useful but feel hidden, confusing, and hard to trust.
EXPECTED IMPACT OF RECOMMENDATION IMPLEMENTATION
Next Steps: A/B TESTING
During our mid-point presentation we learned that the client had identified similar issues as us and were considering solutions along the same lines as what we were recommending. With that in mind, our recommended next steps were to A/B test the feature update variations to understand which changes are garnering the most positive results.
CLIENT PRESENTATION & FEEDBACK
The clients were virtually presented with the research and findings, and there was overall consensus on the key insights we shared. They expressed particular interest and appreciation for the recommendations around improving version-control flexibility and clarifying the more contextual elements of the site that are currently confusing due to jargon.
Next steps for the client are aligned with our first and second recommendations: merging AI tools with the regular editing tools to create a more seamless experience. The research validated their current direction and gave them a clearer roadmap for prioritizing these changes in upcoming design and development cycles.
