Identifying E-Commerce Product Stay & Churn Reasons Using Mixed-method Research
Duration
Dec 2023 - Jan 2024
Methods
User Interviews, Jobs-to-be-done Framework, Screener Survey, Co-design Workshop
Tools
Google Form, Calendly, Google Meet, Excel, Miro, Figjam
Overview
As the founding researcher at Koupon.ai, I established UX research practices from zero and continue to drive proactive and strategic research in the startup company which contributes to reduced risks and increased business profit. In this project, I worked with another researcher Yolanda, and led the end-to-end process from scoping to reporting, utilizing a mixed-methods approach involving interviews and surveys to understand why users adopt, stay or churn our product and collaborate with the cross-functional team to uplift the retention rate.
The Challenge
Koupon.ai offers a website and an app that connects users with Amazon deals and coupons to help them save money on their purchases. First launched in August 2023, Koupon.ai has attracted many new users to try the products and improved significantly through research and iterations. However, the team had no clue about what feature to build in the next step to keep the users engaged. After the initial fast growth during the holiday season, the team was concerned about the low retention rate of the acquired users.
The Process
0. Clarifying the context
The objective seemed clear - to find out what caused the low retention rate. However, as the lead researcher, I wanted to ensure the research goal was truly clear and aligned. Also working at a startup company with limited resources, I needed to prioritize the research that could address the most critical issue of the product. Therefore, when scoping the research, I had several follow-up conversations with the CEO, PM, and designers to understand the full context of the research need and I learned that:
-
The team was eager to understand who were the users and what user needs were met/unmet by Koupon.
-
No significant change was observed - The retention rate had been low since the product launch. This became a burning issue as the increase of new users had slowed down and retaining users became more important.
-
"Low retention rate" was defined by comparing our rate (about 1%) to the industry average (widely believed to be 15%-30%) and there was a big room to improve.
In alignment with the product goal to improve user retention, I reckoned that it was better to look into the problem from two sides: what made people come and stay, as well as what made people leave and churn.
.png)
Therefore, we decided to break the project into two phases. In phase one, we would focus on our power users who adopted our product and had been using it regularly for a while. And in phase two, we would move on to understanding what we failed to do for the churned users. We prioritized the power user study because it was a low-effort, high-impact attempt to keep our happy users and continue to attract other lookalike people.
While the team was hoping to receive user personas as the research artifact, we realized it was not the best option to achieve the research goal and present the insights; instead, I suggested using the Jobs-to-be-done framework to be focused on the underlying user needs and not be distracted or biased by the common peripheral information in personas, such as portrait images, demographics. etc.
Since the JTBD framework was novel to some team members, I did a short introduction of the framework in the weekly team meeting and shared reasons why using this framework would help us gain the most from the research, the team all agreed on this approach.
1. Phase One - Power User Study
Step 1 - User Interviews
To capture the rich context of power users' life situations, goals, needs, and pain points, we chose the good old qualitative method - 1:1 in-depth interviews. We started by defining recruitment criteria, developing a screener survey, distributing the survey, and scheduling interviews. Our target number of participants was 5-6 and we intentionally scheduled 8 so in the case of 3 no-shows we were able to secure 5 interviews and have a good saturation at the end.
Step 2 - Analyze and Categorize Insights
Since Jobs-to-be-done is a defined and structured framework, I argued for a top-down approach to analyze the interview findings. We first broke down and transferred interview transcripts into sticky notes on Miro following the flow of interviews, and then utilized the Jobs-to-be-done "Froce Diagram" to categorize the stickies into Push, Pull, Anxieties, and Inertia to see which contributes to the user adoption of our product and which hindered that.
Cateorization
Step 3 & 4 - Draft and Finalize the Jobs-to-be-done Statements
It seemed that the JTBD statements would just magically come out of the interview insights but that wasn't true. The main challenge we had was to determine how precise/general those jobs should be. On the extremely precise side, you can have a JTBD statement like When I look for deals on electronics, I want to only see those on the newest models so that I can save time and effort" where you have a very specific need under a very specific situation. On the contrary, you can have an extremely broad JTBD statement like "When I shop, I want to save money so I can have money for other things and have a better life" where it seems to be true for everyone and applicable to every product.
After a massive study on the Jobs-to-be-done theory, including books and case studies online, I realized there just wasn't a one-size-fits-all solution to this problem. Every business is different and we have different research goals from using the Jobs-to-be-done theory. Therefore, I led the focus back to our own objectives and our own product. And I set the criteria that the JTBD statements should be precise enough to be unique to our business and broad enough to uncover the root functional and emotional needs of our power users.
This rule effectively guided us to review and modify our JTBD statements to make them really useful to our product.
Step 5 - Prioritize the Jobs-to-be-done Statements
At this stage, we had over 20 Jobs-to-be-done items derived from the research. Not only would they be a huge information overload to our stakeholders but not all of them were of the same importance to our product. Therefore, we reviewed each JTBD statement based on two factors - salience to users and impact on product, and we prioritized those that showed more salience to users from the interview findings and those we anticipated to have a higher and more immediate impact on the product.
Bonus Step - Co-design Session
In my research philosophy, the value of research heavily depends on what stakeholders can take away from it and what they decide to do based on it. So in this project, not only did we create an easy-to-follow presentation, but we also brought the team straight into a 2-hour co-design session where we engaged the CEO, PM, and designers to share thoughts on research insights, and based on each JTBD statement we brainstormed. By doing this, we came up with 14 feasible ideas that were added to the product roadmap, which would improve our product to grow and retain power users.
I created a FigJam board for the co-design session as it offered easier collaboration with designers who used Figam.