{If you missed the first two parts of Risk Management at Scale, read those first and then come back here}.
If you’ve made it this far, kudos on your drive towards reducing risk. Setting up quantitative and qualitative measurements (Part 1), and the process of collecting data (Part 2) are the prerequisites for the real work: analyzing the data. Well, there’s one last step between collecting and analyzing… and that’s physically getting the data.
Many early-stage companies have a limited suite of tools for data export, collection and analysis. You may also receive data in a basic format, like a spreadsheet, SQL export, or a CSV from your internal database. An engineer or data scientist may be able to give you a massively sliced export of the entire database, but it’s often beneficial to keep things simple. Use the software tools you’re comfortable with for this first pass, and keep in mind what additional data you would want next time.
Okay, data is exported and you’re ready to go. But wait, are you the right person to start digging into the analysis? Do you know how to read between the lines? Do you have a clear performance indicator to keep in mind while you are researching? It’s okay if you don’t know right now, but it’s not worth your time if you start exploring without a destination in mind. Take a moment to understand your larger business problems to find those KPIs. Imagine you found a treasure map but there’s no big X indicating where it’s buried. Not much help, is it?
Talk with your team and figure out the problems you’re trying to solve. You can ask everyone what they think is important and use that as a guide… OR go in uninfluenced and see for yourself what is important. Both options have their pros/cons so proceed with what you think makes sense. (Note: If you choose to go in with a hypothesis, a great place to pre-investigate would be your post-mortems. See the post-script for more information).
Let’s use a basic scenario: you have a CSV, are using Excel, and have some moderate skills with filtering / pivoting / vlookups. As a personal preference, we recommend keeping your raw data untouched and pristine in its own worksheet, and then copy-paste everything to a new tab to do your filtering. This ensures you can always return to the full set when you want to take a different approach.
Sort, filter, sort, filter. As you start to poke around, you’ll end up either finding a) the needle in the haystack → outliers, or b) a haystack → trends. Trends are the 80% you should focus on, tackling the bulk of your customers first. For the most part, a single outlier doesn’t tell you anything useful for creating new frameworks to prevent risk. It would be hard to build a system to prevent churn for the one customer that decided to pivot from selling iced tea to adopting blockchain technology (true story).
Look for the haystack, or rather the multiple small haystacks that can be indicators. (If we use Google Search as an example, the big haystack is ad revenue, and the small haystacks are time on site, active users, searches per day, etc). And make sure to document your filters / sorting, so you know what your dataset contains. If someone else with the same data set did the same analysis with the same filters, they should arrive at the same conclusion.
Not all of your discovered trends will be groundbreaking, but document them anyway. Each trend can be an individual clue to unlock a bigger theme within the data. And while you may find a few outliers that match up, don’t get distracted trying to build a case of outliers. Remember: focus on the 80%.
How do you know what’s important? A good starting off point is assume that everything is important! The discovery of “Our customers like it when we make them money” is obvious, so dig deeper. Do they want a 5x ROI, where they pay you $1 a month, and want to make at least $5? Is it more or less? Are there qualitative deliverables that don’t have direct revenue attached to it but can keep a 1.1x customer around for years? Finally, strive for finding trends and causes, not coincidences. It’s critical to understand whether a data point actually drives retention and customer happiness or if the impact is created by another factor.
Customer example: After a thorough analysis, we drew a line on what an acceptable ROI multiplier was for all client tiers. Then, we noticed a second data point: uneven contract vs service levels. Customers would sign contracts for a certain number of widgets per month, and even if they were able to exceed their monthly revenue goals with 10% fewer widgets, they still felt ‘cheated’ out of their full order. This feeling was mitigated going forward by having CS assessing and clearly documenting contract obligations and promises. It also became a helpful data point to cross-examine customer risk. From this, we were able to build the right framework (see Part 4!).
Finally, get a second and third set of eyes on your findings. For person #2, it’s best to have someone who knows the customers, the space, and are preferably close to your team. They are there to help you see the forest, as you’ve spent so much time in the trees. For person #3, go to another department. Their viewpoint is helpful as they can approach it from the product, marketing, or sales mindset. They may also have knowledge you weren’t privy to (i.e. an email blast went out on a certain day and that caused a huge lift in traffic and server costs) that can help color your findings in a new light.
…
Analysis is complete. Now, we’ll learn how to leverage your data to prevent risk (Part 4) and do it consistently at scale (Part 5). To get updates when we publish the additional parts of this series, be sure to follow Sandpoint Consulting on LinkedIn.
For more information about Risk Management, or to request a customized Risk Management Workshop for your team, send us a note at contact@sandpoint.io.
...
POST-SCRIPT
post·mor·tem / pōs(t)-ˈmȯr-təm / noun: a process, usually performed at the conclusion of a project, to determine and analyze elements of the project that were successful or unsuccessful
If your relationship with a customer has concluded, chances are it was unsuccessful, since they decided to stop using your product. Your team should be collecting post-mortems for every churned customer. More information can be found in this blog post.