Metrics Never Speak for Themselves
How we interpret the data all around us is critical to achieving our goals and maintaining our equilibrium.
Imagine you're a campaign strategist for a U.S. presidential candidate.
The horror.
You pull up the latest New York Times polling headlines and discover that your candidate is neck and neck with her opponent. You consider what you could do to move the poll results just a couple of points. You make a plan and implement it.
A few weeks later, the polls show your boss squarely in the lead, even outside the margin of error. Huzzah, you cry! Election day quickly follows, and as the results come in, your candidate is quickly accumulating an impressive vote total nationwide. There's only one problem: the other candidate is accumulating far more Electoral College votes.
Oops.
You realize you're quite possibly the worst campaign strategist ever to hold the job. You forgot that national performance doesn't matter at all. The polls could show your candidate up by 25% nationwide, but what actually matters is that your candidate is down in key states. The presidential election isn't really a nationwide election; it's a state-by-state election.
Luckily, real campaign strategists have this figured out. They know what data they need to pay attention to and how that paints a picture of something that loosely resembles reality if they squint really hard. Even pollsters are presenting us with much more useful data than they did even eight years ago. Much of the engaged electorate also knows to pay more attention to state polling than national polling.
However, when media outlets, from The New York Times to CNN, put national polling data front and center, it's hard to remember what matters. We struggle to narrativize that data in a useful way. The data that's most visible and easiest to comprehend becomes the data that shapes our stories about what's happening in the race.
This problem isn't limited to politics, of course.
Keep reading or listen on the What Works podcast.
We're regularly led astray by the ubiquitous metrics and data that mediate our lives and work.
Data appears to be an objective representation of reality. It seems to promise that, if only we could fully grasp its secrets, we’d have the answers to all our questions.1
When we look at our website analytics, the metrics on the last email broadcast we sent, or the insights on a social media post, are we really staring down the answer to the ultimate questions of life, the universe, and everything? It’s a nice idea—that the perfect strategy is ours as soon as we unlock the cipher. A nice but false idea.
While our unprecedented access to data can help us learn about ourselves, our bodies, our businesses, and more, our data are not objective measures of reality. The numbers and shapes we see in any given dashboard present themselves as they do because a human decided to put them there. That human inevitably has different needs, values, and relationships with systems than we do. They have a set of assumptions about what our goals are and what information would be useful. All of these considerations influence the data we have access to.
Data mediate how we see the world, what we deem important, and how we choose our actions. But we don't see data as media. We see data as facts. Since we don't see data as media, we don't perceive how data mediate our relationship with the world. We don't see the power structures embedded within data media. We rarely bother to seek out the message in data media.
Data are never just data—they are the product of our fears, our hopes, our questions and curiosities. Data are nothing without a narrative to go with them.
Data media shape how we think about the relationship between actions and goals. But data media can also shape our ideas of success, trigger anxiety, and lead us astray. The meaning of the data we pay attention to is always contingent, and it's up to us to make sense of that contingency. Because data mediate how we perceive and act on the world around us, we need a robust framework for interpreting and narrativizing data—especially when it comes to the metrics that are most visible and enticing.
In the rest of this piece, I'll offer three overlapping lenses through which we can view data: predictability, relevance, and actionability.
What is predictable?
One place to start examining the contingent meaning of data is to look at leading versus lagging indicators.
The vast majority of the goals we care most about—things like positive impact on clients or operationalizing our values—aren’t quantitatively measurable. These goals are too complex and nuanced to be reduced to a single data point. However, data do help us discern patterns and judge future outcomes.
So we feel around for things that are quantitively measurable and related to our most important goals. Often these measurements help us predict what’s going to happen and behave accordingly, or they help us take stock of what’s already occurred and analyze it. Predictive metrics are leading indicators. Metrics that tell us what’s already happened are lagging indicators.
Ideally, leading indicators are closely tied to our goals, with enough wiggle room to make adjustments if they aren’t leading us where we want to go. Lagging indicators often represent aspects of our goals without accounting for their full depth and breadth. Whether a particular metric is a leading or lagging indicator depends on the goal we’re trying to apply it to.
Of course, some metrics are just junk. National polling for the U.S. presidential election is a metric that only matters to the media. The national popular vote has zero bearing on who becomes the president. National polling only matters in the sense that it helps the media make headlines, which create narratives, which form expectations, which impact behavior. However, national polls are not leading indicators in any meaningful way. Polls in key states, on the other hand, may be leading indicators of how the election will go.
Because leading indicators allow us to adjust behavior when we're not making the progress we want to make, it's critical that we only give leading indicator status to metrics that have a bearing on outcomes. It's also critical that we pay attention to the right outcomes—often easier said than done. National polling seems like it should be a leading indicator given the way the media covers it. It feels important. So we start to give national polling leading indicator status and, in doing so, give the status of the desired outcome to winning the popular vote. But if that's the focus of a campaign strategy, that campaign may very well lose.
Keeping your eye on the prize might seem like a no-brainer. However, the way we process metrics and data can often lead us to focus on outcomes we have no interest in outside of their relationship with easily trackable metrics. We substitute visible metrics for ones that are more closely related to our goal—and therefore offer a higher potential for predicting our chances of success.
Predictability hinges on the distance between the data and the desired outcome.
Many years ago, a coach encouraged me to consider my ‘earnings per lead’ as a way of thinking about how to hit my revenue target. Revenue is a lagging indicator, of course. Once I knew my revenue for a period, I couldn't do anything to change it. But thinking about revenue (i.e., earnings) in relation to leads (in this case, email subscribers) should have helped me make decisions about growing my audience and making offers.
Leads—as one expects—are typically leading indicators. For the uninitiated, a ‘lead’ means a potential customer or client by virtue of some action they’ve taken. Some leads carry more meaning than others. A ‘hot lead’ might be someone who has verbally committed to working with me or signed up for an initial consultation. A cooler lead is someone who has expressed interest (e.g., attended a webinar or signed up for an email list) but isn’t on the verge of buying yet.
By measuring ‘earnings per lead’ (EPL) in relation to my email list as a whole, I created a close connection between email subscribers and clients even as that connection became more tenuous. There’s a big gap between the action of subscribing and the action of buying. That gap can be even more significant when the call to action for signing up for a list isn’t directly related to whatever offer will be pitched later on.
I ignored the contingency embedded in this metric—the fact that my EPL would change as my strategy changed, that the relationship between my email list and my ability to predict how many people would buy was borked. I couldn't assume that my EPL would remain stable as my list grew—but I did. For my EPL to stay consistent, I needed to bring in leads of the same quality I had attracted before. Unfortunately, my assumptions made that nearly impossible.
How I used the data mediated my approach to list-building and attracting leads. Because the number of leads seemed to be the leading indicator of progress toward my revenue target, I began to focus on generating more leads rather than attracting quality leads. The quantity of leads quickly superseded my revenue target as the desired outcome. Instead of working to increase leads in a way that would lead to my revenue target, I worked to increase leads full-stop.
In this, I was quite successful. My email list grew and grew. My revenue did not.
'Earnings per lead' is a valuable metric. But only if it's utilized to understand the quality of your leads and the value of your offers. If you use it to inspire a focus on the quantity of leads, it's all but meaningless.
It’s important to note here that predictability does not mean causation. A metric can help us predict outcomes without the subject it measures causing the outcome we’re looking for.
What is relevant?
Leading indicators are correlated to desired outcomes. Correlations aren't necessarily relevant, though.
Again, consider national polling in a presidential election. Before the 2000 U.S. election, national polling was consistently correlated to the final outcome. No candidate who won the popular vote failed to win the Electoral College vote between 1888 and 1999. But it's happened twice since—2000 and 2016. It could happen again this year. While it seemed like national polls were relevant because they had been correlated, the two most shocking US elections of the 21st century reminded us that national polls are not relevant, even if they are correlated.
Similarly, at one time, the number of leads I attracted was positively correlated and highly relevant to my revenue. Once I started focusing on quantity over quality, though, the correlation remained, but its overall relevance plummeted. Sure, more leads would result in more revenue, but the relationship was no longer stable. I had less and less confidence in it.
Relevance is a product of the number of variables in the relationship between a leading indicator and a desired outcome. The further the distance between the leading indicator and the desired outcome, the more variables are bound to be present. The more variables there are, the less an indicator should be considered predictive—in other words, the more variables there are, the less likely a metric is to be a leading indicator at all.
The greater the distance between the data and the goal, the less relevant that data will be.
When I focused on the quantity of leads rather than the quality of leads, I widened the distance between my leading indicator and my desired outcome. I could achieve a quantity of leads by offering free events and downloads that I knew people were likely to take advantage of. Without ensuring that those events and downloads were targeted at highly motivated potential buyers, I introduced a host of new variables: urgency, readiness, pain points, goals, etc. What was once a straight line between leads and buyers became a wiggly, broken, unpredictable line.
None of my list-building tactics were nefarious. They just weren't very strategic. My failure to hit my revenue target that year, despite strong email list growth, was my own fault. My coach didn't tell me to pursue quantity over quality. There weren't any user interfaces unduly influencing my tactics. I just messed up because I forgot to prioritize the relevance of my data to my desired outcomes.
As I mentioned, this was a long time ago—almost a decade. Things have changed quite a bit since then. Today, platforms and apps love to help us track our metrics without regard to their relevance or connection to our desired outcomes. When I log into my podcast host, I'm presented with a 7-day chart of downloads. The metrics I see when I log into Substack are more relevant, but they're still predicated Substack assuming what my desired outcomes are. My "professional dashboard" on Instagram offers "insights" such as views, interactions, and followers—none of which have any bearing on my desired outcomes.
These dashboards are a prime way that data mediate our perception of reality.
Data as media shape how we perceive the relevance of metrics like podcast downloads, Substack subscribers, and views of Instagram posts. Data as media hide—intentionally or not—the variables that inevitably complicate the information available. Data as media draw the curtain between strategic action and the people who choose what metrics influence that action.
What is actionable?
Just because we have a data point doesn't mean we can (or should) do anything with it.
Let's look at presidential polling one more time. Last week, The New York Times released a poll showing Vice President Harris up 50%-46% over Trump in Pennsylvania. This poll result was an outlier—both from other PA polling and from national polling. If one were to view this poll in isolation, it might suggest that PA was in the bag. They could pull ad spending and put it into a state where the margin was closer.
Of course, they're not doing that because an outlier result isn't an actionable data point. That doesn't mean it's wrong; it means it's not useful.
For data to be actionable, it must be supported by a pattern. Patterns further mediate data and illuminate the relationship between data and our desired outcome over time. Without a pattern, there are too many variables and too little predictability to make informed decisions about what to do next.
When data forms a pattern, we have a theory—a narrative—we can test based on our next actions. If we can influence the pattern, then we know how our actions relate to the data and have further proof of our theory. If we can't influence the pattern, we must rethink our theory.
In my 'earnings per lead' example, I didn't have a pattern. I had a single data point—my previous year's revenue divided by the number of people on my email list. If I'd looked at my EPL for previous years, my guess is that it would have been higher. I might have discovered that the pattern was a downward trend. Instead of focusing on growing my list, I might have asked myself how I could improve the quality of my leads or make offers better suited to the people on my list.
Instead, I took action on the single data point and, by the end of the year, wondered how things had gone so wrong.
Patterns are powerful tools for navigating the distance between data and a goal.
Patterns help us understand the contingencies embedded in our data, and they also help us draw closer, more direct connections between data and our goals. A pattern might not shrink the distance, but it will make the trip easier to navigate.
The Case for Data Literacy
Much has been said about the necessity of media literacy in the age of truthiness and fake news. We need to learn how media can be manipulated, and trust can be misplaced. We must all consume news and other information responsibly if we're to maintain a functioning democracy and economy.
Similarly, systems literacy is critical for understanding the relationships between people, resources, and ideas. The more we attune our senses to perceive how systems reinforce each other, the more effective we'll be at catalyzing change and resisting the status quo.
We also need data literacy. That doesn't mean knowing how to read a spreadsheet or run the crosstabs on a research survey—although both are helpful. Data literacy requires us to acknowledge that the numbers don't speak for themselves, that statistics lie all the time, and that just because you can measure something doesn't mean it should be managed.
There's more to data literacy than what I've outlined here. But beginning with a critical framework of predictability, relevance, and actionability is a good place to start. Adopting this framework will help us make smarter decisions and take more strategic action all while allowing us to ignore the data that are just distractions and nothing more.
This piece draws heavily on C. Thi Nguyen’s concept of value capture and Byung-Chul Han’s critique of data and information superseding narration and theory.