Oct 2025Samantha Speights
Figma Make Experiement
Methodology
Structured A/B test pitting an AI-assisted group against a manual control group across three realistic design tasks
17 UX participants total 11 in the AI group, 6 in the manual control group
Participants self-timed each task to measure design speed and efficiency
Designs evaluated by a DesignOps team on technical quality and by a panel of Principal/Manager-level designers on UX quality using blind grading
Prompt log analysis conducted using an agentic AI tool built in Google Gemini to evaluate how participants interacted with Figma Make
Results synthesized across four evaluation methods: time savings, output quality, prompt log analysis, and usability testing
Hypothesis
Figma AI features can save time in UX design tasks while maintaining or improving design quality. Designers using Figma Make will reduce time spent on UI design by over 50% compared to a manual workflow, without a significant loss in output quality.
My Role
I led the design and creation of the prompt analyzer, an agentic AI tool built in Google Gemini that evaluated participant prompt conversations against a structured quality rubric. I also supported the overall research effort and contributed to the final readout delivered to EUX leadership.
Prompt Analyzer
Quality & Time Validation
an agentic AI tool built in Google Gemini that evaluated each participant's Figma Make conversation against a structured quality rubric, scoring interactions from 1–5 across six categories. It was designed to surface patterns in how designers communicated with AI and identify what made certain prompt conversations more effective than others.
Rubric Description
The rubric evaluated the overall quality of each participant's AI prompt conversation from start to finish, assessing both the clarity and effectiveness of how the designer communicated with Figma Make. It was designed to capture not just what was produced, but how well the designer and AI collaborated to get there.
Categories
Clarity Throughout Conversation
Relevance & Focus
AI Responsiveness
Efficiency
Collaboration
Outcome Alignment