3 Comments
Dec 28, 2022Liked by Robert Yi šŸ³

Just gonna jump in and pile on the hot-takes here...

First off, I donā€™t know anything about hyperquery so Iā€™m staying out of that part of it.

But analysis both is and isnā€™t development.

Analysts write code, it should be repeatable, and correct. In other words, it is development. There is nothing fundamentally incompatible or suboptimal with using an IDE for this. Depending on the IDE of course.

But analysts donā€™t ship code. Not the same way. I mean, you could set up a CI system to generate and publish a report on merges, but thatā€™s quite fanatic. I tried. It was short-lived. So we default to think ā€œnotebookā€ as the solution because itā€™s code but has plots and isnā€™t an IDE.

RStudio is an interesting case study here. It is unquestionably an IDE. I have shown it to colleagues in IT and they all agree it is an IDE. It has all the debugging features. The git integration. The testing framework integrations. But it is also definitely a tool for analysis. It displays plots. It produces reports. It publishes interactive visualizations. It can even help you write to word if you need to.

So where does this leave us? A lot of IDEs arenā€™t good IDEs for analysts. DataGrip (or whatever other SQL client) isnā€™t, and afaik never attempted to be. Same with IntelliJ, even though someone has tried to duct-tape a ā€œshow plotsā€ feature onto it. But there are IDEs for analysis. In addition to RStudio, there is DataSpell (which I havenā€™t used in a while, and had a very ā€œnotebook with stuff glued onā€ feeling to it if memory serves). Most seem to be python-first though. SQL is left behind.

I have left out any discussion of new-ish entrants like hex (and hyperquery) because I donā€™t know enough about them. But It feels like we are at a Henry Ford kind of moment where everyone is so used to notebooks, it is the only thing they can think to ask for. The next winner in this space introduces something new that is neither a notebook nor IntelliJ. But they will have to teach people how to use it.

Expand full comment