User Experience Optimization
via RealEvals
We combine the rigor of quant and the depth of qual to provide immediate feedback on real experiences of real people. This is a combination of observed interaction, reported sentiment and comparison vs. competition. The outcome provides clear, detailed input for product, app, model optimization.
Working With F'inn:
Traditional Research:
A/B Testing
Real life testing of apps, devices or recreation of experiences
Mixed methods in sequential fashion with long timelines and high costs
Intertwined mixed methods, continually providing insight and generating new questions
Wait til prototype stage, after many decisions are locked
From experiment through public launch
Project-based
Continuous feedback during development to build a knowledge base and benchmarks
Our Way of Working
Role of AI
This whole method and program is built as input for AI development. Behind the scenes, we are utilizing AI for analysis, coding, and transcription of the interactions between the user and AI, all which help us expedite the process.
We continue to experiment with ways to bring AI into the process, yet will only do so when we are confident the synthetic respondents will accurately represent real people’s real experiences. Based on our work with synthetic respondents, the level of accuracy / predictability is not yet ‘good enough’ by our standards.
AI moderation is being trialed, and will be scaled consciously as the capabilities evolve.
People First
We built the RealEval program together with our clients to focus on:
-
Real users vs. annotators, trained evaluators or those who know the technology
-
Real experiences vs. lab-based or prompted experiences
-
Real interactions, between users and AI.
Users interact with AI / apps / products as they naturally would. We observe their behavior and responses while also gathering concrete quantitative feedback. We often test iterative models with the same participants to understand progression or regression; or comparative models to determine how to win vs. competition. This feedback is utilized by Engineering to optimize based on what people want and enjoy, their JTBD and the issues they experienced.
Case Study:
The Impact
The RealEvals program has become an integral tool for model optimization. The outputs range from overarching frameworks for development to hyper specific insights into the UX. Each eval is customized to meet the moment for the development team.
Our Approach
This ongoing program incorporates:
5
IDIs for deeper insight around specific experiences or users
4
Competitive comparison via captured interactions and full transcripts
3
Use case identification and specific analysis
2
Establishment of thresholds to determine market / launch readiness
1
Iterative feedback on each model generation via custom virtual activities
The Ask
Engineers needed a fast way to test pre-release models with real users to gather feedback reflective of in-market response.
