Gaming your Metrics
Lessons in Typing
We all want to be data driven, but "as soon as a measure becomes a target, it ceases to be a good measure." This is known as Goodhart's law, and describes the paradox of setting goals for a growing business (or any field really). The problem with metrics emerges when the system begins to rebuild itself around an optimal path to hit that metric.
I first remember experiencing this with a really terribly written typing program for the Apple II in a computer lab when I was third grade. One of the key metrics with typing is words per minute. When you're trying to build high performance typers this is great, you can judge yourself against your peers or yourself. Unfortunately, the teaching software didn't evaluate accuracy as part of lesson progression. I realized by not typing real words and just hitting spacebar a bunch I was hitting 70-80 (maybe 120) words per minute I could blow through the course. The software had me classed with the top typists in the world.
When I decided that I should actually learn typing, the software saw my drop in metrics and decided I must have not mastered that particular module so I was forced to repeat "q" "u" words for the rest of our computer lab time. I didn't learn how to type until I was in high school.
Find Balance
Metrics should have tension with each other in order to achieve a balanced outcome. Words per minute can be paired with accuracy, that way success at one is not able to borrow from the poor performance of another. The sign of a good metric pairing is when one metric moves negatively when the other moves positively and they are strongly correlated.
Let’s take a quick look at one of the top metrics for any business, Recurring Revenue. You can easily grow unhealthy systems using only revenue as a metric. Sales folks may promise products that don’t exist leading to frustration and anger from your customers and employees can be driven to work long hours or quit because of the unhealthy environment that has emerged. Both of these can be measured as churn (or as I’ve heard it described “reverse revenue”).
One super simple measure of customer satisfaction is the Net Promoter Score. This is the reason behind all of the emails you get asking “Would you recommend product X to a friend?” The system classifies users as promoters, passive and detractors and can be a strong indicator of if people are actively evangelizing what you’re doing or if they’re unhappy and looking elsewhere for something to solve their problems.
Pairing these two metrics together can indicate when you’re pushing too hard on revenue and impacting customer satisfaction. If you go a step further you can also get insights on when employees are being pushed too hard and business growth is impacting their satisfaction. Keeping a close eye on both will ensure that the system is not borrowing against future growth to hit the short term numbers.
In a world…
A bit of a tangent, I get a lot of joy from video games where they take a small concept and explode it out to the most important thing that’s driving everyone in that world. One of my favorite examples of this was a Switch game called “Golf Story”. You’re an adventurer in a world that is obsessed with golf. There’s no indication that anyone has any other jobs or hobbies and everyone in this world is 100% focused on moving up the golf rankings. There are disk golf gangs that frown on traditional golf and classy golf elites.
Let’s take our metrics to that same extreme to think through the unintended consequences before subjecting everyone to a singular goal of hitting that one number.
What does something taken to that kind of extreme look like?
How does it break down and fall apart?
What sort of cliques form in this new metric world and push back or embrace the system?
This thought exercise can shine a light on weaknesses and help identify some counter metrics to achieve balance. It might even be a world we want to live and work in.