Analysis or dashboard?

Let's talk about the Pareto law, the dashboard fallacy, and how to answer the hardest question in analytics

👋 Hello! I’m Robert, CPO of Hyperquery and former data scientist + analyst. Welcome to Win With Data, where we talk weekly about maximizing the impact of data. As always, find me on LinkedIn or Twitter — I’m always happy to chat. 🙂

And one more quick announcement: our Hyperquery 100 program is still ongoing, giving you access to a lifetime single license Team membership for $100. Sign up if you haven’t already!❤️

---

It’s been nearly 2 years since Seth Rosen’s trashboard tweet, and yet the trashboard epidemic is still quite real. We still encounter companies that are curiously happy with having thousands of dashboards, their analysts left to spelunk deep caverns of ad hoc requests with naught but an IDE.

There are good intentions all around, of course—ad hoc work can be painful, and an automated solution sounds appealing. What’s more, dashboarding is a safe bet — we still live in a world where the dashboard is still accepted as The Default Tool For Analytics, after all.

Let’s talk about how we got here, why this needs to change, and where we should go from here. In short, in what follows I’ll argue that:

• Decades of heritage have brought us to a weird place where dashboards have primacy.

• But dashboards are not always the best way to provide value to your org, especially after the first few.

• If you believe the two statements above, your beliefs, your processes, your tooling need to fundamentally change.

Midjourney’s a beast, ain’t it?

The problem: dashboard primacy in a Pareto world

Dashboards vs. analyses, by way of cooking

So don’t misunderstand — dashboards are fantastic, they’re just blunt. They can serve endless needs, but in doing so, rarely do they serve particular needs perfectly. It’s the Heisenberg uncertainty principle for data: the more data you expose, the less sharp your insight. Dashboards, by nature, prioritize data exposure. They offer data by itself, and interpretation is left to the consumer. A certain number of core dashboards are certainly required, of course. Every business unit has core metrics and events that it tracks, and these need to be surfaced somewhere — the dashboard here acts as the heartbeat of the business. But beyond this, value greatly diminishes after each nth dashboard.

Analyses, on the other hand, are sharp. They expose interpretation, enabling more precise relevance to business problems. If dashboards are like salad bars, analyses are made-to-order meals. And just as specific meals fit cravings better, so too will analyses always fit specific business needs better. On the other hand, incremental salad bar items (more dashboards) have diminishing returns — they’ll never quite satisfy a particular craving¹, but they’ll crudely solve most hunger-related problems. You can benefit greatly from having a few, but you don’t need to continue building them endlessly.²

Then why do dashboards still have primacy?

Although I’ve found folks generally agree on that sentiment, we live in a peculiar time in history where dashboarding still retains its primacy. It made sense when data was slow, data was hard to handle, and dashboards were cumbersome to make — core dashboard maintenance was the entirety of an analyst’s job. But as the warehouse sped up, as analytics needs changed, as the returns on additional dashboards diminished, our mindset never adjusted accordingly.

We’ve somehow become trapped by some perversion of the sunk cost fallacy: we’d allocated all incremental resources to dashboarding, and so we started to believe dashboards were of the utmost importance.

I’ll concede that there’s an inherent attractiveness to dashboards: dashboards allow us to build with high coverage in mind. They sell the dream that our ad hoc request volume will drop to 0 one day. One day, given enough dashboards, we’ll have addressed all questions and all follow-up questions that are to come.

But unfortunately, running a business is rarely so formulaic. Power law distributions proliferate. Preferential attachment begets pockets of value and worthlessness, change and stagnation, accumulation and void. Not all things are equal. It’s the world of the Pareto principle, where avalanches drive change. Where 20% of the effort drives 80% of the outcomes.

And in this world, analyses are no exception. If you were to rank order all the work you’ve done by impact³, I suspect you’ll find most of your value add came from just a few key analyses. And in this world, the dashboard flounders — what we gain in exhaustiveness, we rob from the avalanche.

Where do we go from here?

I’ve long been a proponent of elevating the role of the analyst. But I seldom do analytics work these days, and when I recently got my first ad hoc request in months, I’ll admit I jumped directly into the data. I had a strong urge to answer the question as quickly as possible — I viewed the work as a hindrance, not an opportunity. It was an attempt to diminish the work required of me, but in doing so, I fell from Co-founder to human API. I fell victim to the bad loop ruining analytics, even as a self-proclaimed curate of this borderline analytics religion.

"Everything is a self portrait. Everything is a diary. When you care about something it becomes fairly obvious to everyone else. When you care, your devotion or discipline is directional. Most people can intuitively feel it."
- Chuck Palahniuk

Ergo: we need to change our defaults — our beliefs, our processes, our tools — or risk falling back into the same old habits.

Belief: value beyond data pulls.

The first step in driving any sort of behavior change is to merely internalize that there’s a clear reason why you’re doing what you’re doing. We need to believe that our value add extends beyond getting data. By all means, if you believe that your organization’s primary need is a core set of dashboards, then set those up and focus your efforts on that. But if you have an inkling that you can provide leverage outside of this — by helping folks make better decisions, or by even keeping others intellectually honest in the conversions—then internalize this and proceed under the assumption that it is true.

Processes: analysis-by-default, not dashboard-by-default.

To some extent, I believe we all want to have more impactful roles, drive more business value, participate in higher levels of strategic problem-solving. But beyond simply recognizing that, the remaining hurdles are still substantial. In my estimation, the most debilitating hindrance henceforth is that it’s far too easy to fall into bad habits. We need to build systems that make it easy to do the right things and harder to do the wrong ones.

“Goals are good for setting a direction, but systems are best for making progress.”
- James Clear, Atomic Habits

I’d start by considering a move to an analysis-by-default world, particularly if you’ve already invested heavily into self-service systems and a set of core dashboards. If we spend our IC time trying to maximize the usefulness of our work (analysis), rather than trying to maximize the scalability of our work (dashboards), it follows then that our work will end up being more useful, by definition.

Beyond that, build templates (here’s mine), standards around how work should be done. Write up hardened operating principles for your team. Hell, set borderline toxic policies to enforce better behavior: analyses don’t count unless you put them in our centralized, agreed-upon place, for instance. Adopt rituals that reinforce good behavior: have daily stand-ups where you present this work; bake ritual compliance into your performance reviews. Do whatever it takes to make sure folks do the things that you know will make them better.

Tools: notebook? At minimum, an analysis-first workflow.

If we believe in analysis-by-default, we need a set of tools that inverts common working patterns and pushes us to build analyses first, then dashboards, not vice versa. Certainly I’m biased — this has been the philosophy behind building Hyperquery — but even if you’re not sold on our vision, even ensuring you don’t jump into Tableau for every analysis is a worthy first step. At minimum, narrative-first is a must. By simply forcing yourself to default to words as deliverables, rather than data, you’re already breaking out of a ruinous habit.

Narrative-first forces alignment, relevance, value over immediacy. Narrative-first elevates the conversation beyond technical contributions to strategic contributions. Narrative-first nudges to your stakeholders that your value lies in narratives, not code.

Final comments

Analytics is ripe for change. Even if you don’t buy the arguments I’ve laid out, it’s undeniable that our industry has changed astronomically in the last decade. And with new cars come new roads, new laws. At minimum, I hope this post pushes you to think about what that new world ought to look like.

1. Unless you explicitly want a salad, but who ever craves a salad?

2. I’ll concede: perhaps your company/domain has a particular business model that does benefit from a million dashboards (and dashboards alone). In that case, more power to you.

3. Okay, okay, nearly impossible, I know, but try.