Dear dbt: all you need is love.
A crude network model of dbt's community stability, and an appeal
đ Hello! Iâm Robert, CPO of Hyperquery and former data scientist + analyst. Welcome to Win With Data, where we talk weekly about maximizing the impact of data. As always, find me on LinkedIn or Twitter â Iâm always happy to chat. đ
I was in high school when Facebook started to kill Myspace. And contrary to the many articles that credit operational excellence or obscure network dynamics, I remember clearly why my friends and I started to make the switch: the social elite â college students â had moved to Facebook. Facebook stole the pillars of the Myspace community, and it wasnât long before the flock followed.
But this is not a post about Facebook. Itâs about dbt. Obviously, dbt is not a social network. But it is undergirded by one â dbtâs core offerings are evangelized through a massive community. And this community is its differentiator, its competitive moat, its lifeblood. And for the first time in the companyâs history, this community is showing cracks. Sentiment has been faltering, and this could be critical for a company whose community is such an integral part of its business.
While it still seems most likely that dbt labs will one day end up as Facebook, capturing and preserving the hearts and minds of the analytics elite, growing dissent from dbtâs most vocal champions has started to smell of Myspace. Letâs talk about whatâs happening, why Iâm still bullish on dbt, and how they might avoid becoming the Myspace of data
.Whatâs happening: community sentiment shifts
dbt has long been the darling of the data community. They filled a glaring need: analysts wanted to build transformations, but views were brittle, schedulers had a high barrier to entry, and traditional BI was vendor-locked. dbt offered a code-first (robust), SQL-first (accessible), open source (not vendor locked) solution, filling a dbt-shaped hole in our toolchain. Analytics engineering was born, and a vibrant community of evangelists formed around it.
But recently, sentiment around dbt has started to cool. A couple months ago I went to Data Council in Austin, and a teetering ambivalence/gloom clouded mentions of dbt. Youâll hear a lot of complaints these days: the cloud IDE is still disappointing, the dbt core experience hasnât gotten much better over the years, the semantic layer turned out to be two goats in a trenchcoat, people noticed serious issues at scale, an unsettling price hike came out of nowhere.
While these issues should certainly be addressed, whatâs odd is that most of these problems are not that new. But the fact that theyâve started to govern the narrative may point to something precarious. Letâs hypothesize what caused this to happen.
A crude network model of dbtâs community
Dbtâs business is structurally bizarre. For most open source companies, the user of the open source tool and the user of the paid offering are one and the same â think Gitlab, Docker, Elasticsearch, Superset. And so, community management is straightforward: make the product better, the community will grow. The flywheel is linear.
dbt, on the other hand, has two distinct products: dbt core (a CLI tool) and their paid offering, dbt Cloud (a cloud IDE). As a result, their community instead has two sub-communities associated with each â cloud and core â that are deeply entangled. Perception and sentiment flow freely between them, meaning product investment into one can have nonlinear, unpredictable effects on the other. Invest in dbt Cloud, and dbt core users might feel shunned. Invest in dbt core, and dbt Cloud users might migrate over, decreasing revenue. Itâs a double pendulum, and so, chaos and nondeterminism reign free. And in a world where product development resources are scarce, this is a product strategy nightmare.
At the present time, Iâd hypothesize that all the product love in this system stems from dbt core evangelists, rather than from the paid offering. As a result, these folks are the primary agents keeping general perception afloat. And this is the root of the problem â the sentiment from these folks has been lukewarm and declining for the last few years. It started with product qualms, but has steadily shifted into this suspicion that dbt âisnât for [them] anymoreâ. Weâve started to hear whispers of âeh, dbt cloud sucksâ or âdonât use the semantic layerâ voiced in dark corners by the most technically competent dbt users.
And so dbt is in a curious state where thereâs little love for dbt cloud, but the folks that are driving perception â the dbt core evangelists â are slowly starting to feel disillusioned. Yet, they need to somehow move to a world where cloud begins to engender love on its own, or risk failing to sufficiently monetize to justify their hefty valuation.
Love is all you need
So what should dbt do? Itâs obvious, isnât it? Make dbt cloud amazing. Continue to build out dbt core so the pillars of the community stay happy. Follow through on the semantic layer. On the one hand, this seems to be exactly what dbt labs is doing â theyâre investing on all fronts with full force. But on the other hand, somehow progress feels marginal.
From a feature-level perspective, the kinds of things I know I want out of dbt are fairly simple: theyâre the same things Iâve wanted out of dbt since I started using dbt back in 2021. I want dbt cloud to open immediately. I want someone to finally admit that ELT was probably a mistake (or at least an overcorrection) and tell me how to go back to having less models. I want dbt build & dbt test to be as close to instantaneous as possible. I want dbt-native Intellisense, everywhere.
Theyâre the flavors of the dream that I think we collectively dreamt when we first executed dbt run
. A dream of a seamless, end-to-end engineering-like experience. The lingering taste of engineering practices Iâve gotten so far isnât enough â I want the rich ecosystem around it too. And yes, sure, now I want a semantic layer, but only because again, I was sold on that dream, and it wove its way into my expectations.
On the one hand, dbt labs has certainly listened. They improved loading times. Theyâre building the new semantic layer. Theyâve built rich Intellisense. Theyâre constantly pushing to help folks control the model chaos. But it still takes ~10 seconds to just get started. And weâre nearly a year out from the first public release of the semantic layer and two years since their first announcement of intent, with nothing to show for it. ELT still stands strong, and all signs point to crowd control, not policy change.
Thereâs a feeling that somehow these solutions didnât quite hit the mark. Pains have been patched, but alongside these came a million things I still donât know if I need, and Iâm still bleeding beneath my bandaids. It feels like theyâre succumbing to some variant of the next feature fallacy, pulled in arbitrary directions in an endless game of whack-a-mole.
Instead, imagine if we instead got step function change along the axes that mattered, rather than broad incrementality against everything. Imagine a dbt cloud that loaded instantly. Imagine if dbt run and dbt test could somehow be executed instantaneously. Imagine a semantic layer that worked locally and on dbt cloud, one that put LookML to shame.
In short, I want to fall in love with dbt again. And while Iâm far flung enough from the internal guts of dbt to understand whether these are possible, these sorts of things donât seem so insurmountableâsome heavy refactors might be necessary, for sure, but Iâd wager theyâre worthwhile. Their present roadmap appears to patch problems, when the issue at stake is this: how can dbt labs keep growing love? And love comes from real change, not lip service.
Final comments
If it wasnât obvious already: Iâm not sure who Iâm writing this for. The dbt product team certainly knows the nuances of their business far better than I do. But like many others who shared their perspectives around dbt over the last year, I felt the need to scream into the dark, because I still love dbt. I just desperately want it to get better. Itâs filled one of the most substantial needs in the market with a tool built with developer experience in mind. And I know I very well may be the boy who cried wolf here rather than the canary in the coal mine. But I hope that this at least gives voice, more depth of character to the user pain that I and, I imagine, others have been silently bearing. And I hope this at minimum gives some small insights to the folks at dbt. At the end of the day, Iâd rather run the risk of sounding redundant than see my favorite data tool turn into Myspace.
That said, Iâm hopeful theyâll be able to have their cake and eat it too. Product priorities at dbt over the last few months have certainly changed in some profound way. In a recent announcement, Tristan announced a renewed focus on dbt core with practical problems in mind. With their acquisition of Transform, thereâs hope that the semantic layer will be salvaged. Dbt cloud, as well, has been making steady improvements to the user experience. And in a recent announcement for dbt core v1.5, Jeremy Cohen did one of the few things dbt could do to win evangelists back coming off of heartbreak after heartbreak â he reminded us of the humanity of their endeavors.
dbt is for people, and people arenât perfect. Weâre going to keep building this imperfect thing, and finding ways to build it together.
And to that: â¤ď¸.
as a high school student. AS A HIGH SCHOOL STUDENT.
Disclaimer: I know little about the inner workings of dbt labs, so for those of you that actually work there or are more intimately involved with them, take this with a heavy dose of salt.