Running a SaaS without rich customer data is like flying blind. Dumb luck, good guesses, and the right product-market fit will only get you so far.
This was exactly the problem I ran into when I was working for an Amazon SaaS company. We had a first mover advantage in the market, but missed some opportunities to get deeper customer data in lieu of paying off long-standing technical debt.
This became a challenge when we acquired a new company and wanted to onboard our current users onto the new platform. Although in theory we had the same customer base for both softwares there were certain restrictions on the new software that meant only a subset of our current customers could use it.
The Balancing Act
As product manager it was my team’s responsibility to get as many people adopting this new software, but there as with any project there was a balancing act between time, scope and cost when delivering.
Time. The market and competitive landscape were changing. Our competitors were becoming more sophisticated and the market was turbulent after a recent Amazon Terms of Service change.
Scope. We were potentially moving thousands of new users onto a platform that had only supported hundreds before, but we didn’t know how many of these thousands of users we could adopt our software.
Cost. There was an influx of capital into the new software acquistion. Despite additional money for resources it still took time for them to be properly trained and working at full capacity.
The simple solution was to open up a beta to start scaling the users on the platform immediately while the product team came up for a non-development solution for the lack of customer data that we could scale as we grew.
Our Beta Metrics that Mattered
In order to make the most of our time and energy we chose to focus in on two metrics for the Product team in order to measure the success of our Beta.
KPI One | Customer Service Tickets
- Leading indicator of any issues with scalability and onboarding, we wanted a way to qualify data as soon as possible. Although an imperfect measurement, it gave us immediate feedback on our user experience.
- We categorized each of the incoming tickets (onboarding, technical issues, general inquiries, disqualified users)
- Product Team dove into get one-on-one time with the customers instead of using the support team to field tickets during the beta.
We knew that we were going to have a mismatch between the invitees and people who could adopt the platform, we just didn’t understand the scope of the problem. Customer service tickets would give us a first hand idea to how well the incremental changes we were making in our onboarding and segmenting were working.
KPI Two | Adoption Rates
- This was a trailing indicator of a successful beta as it took time for users to adopt the product and we wouldn’t know if our efforts were working for up to 2-3 weeks as customers moved through our funnel.
- This was defined as connecting their Amazon account to our software, uploading their data, and logging into at least three times.
- We hypothesized that adoption rates would be inversely correlated to onboarding customer service tickets.
Our Adoption rate calculations were able to give us insight into how many of the invitees were the right fit for our software and of these invitees how many of them were actually adopting our new offering.
This turned out to be a more challenging calculation than we originally anticipated. We ended up using three different calculations to get a better idea about where we were losing customers in our funnel. The KPI’s were focused around different actions and gave us insight into whether to be looking at the invite messaging, segmentation techniques or onboarding (this was done in cooperation with the marketing team).
Opening up the gates to Beta
We started small, adding between 50-100 users a day to the new platform. Using Customer service tickets as a leading indicator, two issues immediately popped up.
- Not all our invitees could use the new platform – certain Amazon users were disqualified because of their fulfilment type, geographical location or the size of their businesses. (Approximately 75% of tickets)
- Users were confused about how to connect their Amazon accounts to Merchant Metrics (approximately 15% of Customer Service tickets)
Ideally, we would just segment the leads that could use the new SaaS offering, but (as previously mentioned) we didn’t have enough rich user data from the parent software to determine who these customers were.
How can we get enough data on our users to properly segment them and maximize adoption rates?
Given previous attempts at surveying our customers we knew that they weren’t motivated to fill out long form surveys and we would miss qualified customers that opted out of taking the survey.
We needed a solution for our users to take the smallest amount of commitment in order to segment themselves.
We needed to get invitees to self-qualify in the smallest way possible
- YesInsights allowed customers to click on a link to answer short one or two question surveys directly in their email or in-app messaging.
- Intercom can send customer and behaviour specific messaging based on tags, segments or other custom customer data.
- Zapier connects applications without formal applications. We used Zapier to tag customers in intercom based on survey answers from YesInsights.
The user workflow looked something like this:
- Invitees would receive invitation email to set up their account and be redirected to a link to set up their password.*
- Once the user set up their password they would be logged into the application
- Here a screen would pop-up asking the user to qualify themselves by clicking links built directly into the in-app messaging:
- Which marketplace they were selling in
- What type of fulfilment channel they were using
- Based on their answers to the one-click survey questions (YesInsights) they would be tagged accordingly (via Zapier) and sent specific instructions on how to proceed either by email or in-app message (in Intercom).
*We decided against sending them a qualifying email with a one-click survey before they set up their account. Although some users may have found this frustrating, it allowed us to capture customer data with intercom and made it easier to activate them when we supported their business.
Here is where we ended up –
We were able to get more data on our users – what size of company, fulfilment channels, geography with minimal friction during the onboarding process. This information was stored in intercom automatically with Zaps. (Under the “Beta” umbrella we were able to expand this into bi-weekly surveys to gather even more specific data on our active customers)
We better understood adoption rates. We were able to figure out who was adopting our product, why they were (or weren’t) and make adjustments that would affect real qualified user adoption rates.
Reduced customer service tickets by over 60% inside of two weeks.
Grew a qualified list of clients that were interested in the product, but were not serviced by the existing offering. When we expanded into Europe we were able to target these customers and reactivate them.
Ideally we would have all the customer data we needed, a scalable platform and we could just move the right people onto the platform and all pat ourselves on the back.
But Start-Ups are messy.
It is unlikely as a Product Manager you will ever inherit a perfectly automated, documented product without any time or cost restraints.
In this case we were paying off debt not from our SaaS, but from our parent software. Instead of griping about the lack of data, we found innovative ways to get “good enough” data to make decisions about our launch today while managing our need for moving to market quickly.
We made smart, long term decisions despite being under incredible pressure from management to onboard and scale as quickly as possible. My team worked hard to set up feedback systems along side our segmentation and onboarding systems and still make deadlines.
In the short term this meant some long hours, but it also meant that we were able to build a system that we could pinpoint where customers were leaking out of our funnel right now. This could have been accomplished with fewer man hours but it would have lacked the data and the feedback system we implemented along side it.
The automated feedback system that provided a constant stream of customer insights. This helped us form good relationships with early adopters and power users, gave us a treasure trove of information when it came to building out a longer term roadmap and systemized our information structure across the entire project.