An interesting topic that, to me, is nothing new. As a BI engineer something I have worked since I started. The only difference is that it is now ripped out of the BI tool and added additional features. Interesting to see the Evolution of the Semantic Layer 📈 :
1991: SAP BusinessObjects Universe and BI semantic layer
2008: Master Data Management (MDM) (with MDS from Microsoft in 2008)
2013: Kimball discussed the concept of a semantic layer in #158 Making Sense of the Semantic Layer
2016: Maturing BI tools with an integrated semantic layer such as Tableau, TARGIT, PowerBI, Apache Superset, etc. have their own metrics layer definition
2018: Jinja templates and dbt eroding the transformation layer into a semantic layer
2019: Looker and LookML popularized as the first real semantic layer
2022: Modern Semantic Layer, Metric Layer or Headless BI tools such as MetriQL, MetricFlow, Minerva, dbt arose with the explosion of data tools (BI tools, notebooks, spreadsheets, machine learning models, data apps, reverse ETL, …)
Agreed dbt's metrics are quite limiting, but if you squint your eyes, there isn't really much stopping dbt from adding in foreign key relationships and removing a lot of those limits. There's some added friction in that it takes a lot more boilerplate to get a metric down, but this does have its benefits in giving more weight to the metric itself (over the rest of the semantic layer components).
At the end of the day, my big question is: will the power of dbt/looker's integrations/momentum win out? Or will something with a different (maybe better) user experience win (like GoodData, but to be fair, cube or malloy or some other startup)? Is this a game of integrations/momentum or a game of user experience? At this point, I'm not sure, but it's generally really hard to bet against momentum
I think that the most crucial thing for metrics together with the semantic layer is standardisation. It would be so much easier if there was a standardised open-source semantic layer that would be utilised and extended by other vendors (dbt, Looker, GoodData, cube, etc.). It would solve many things, such as vendor lock, features portability (implementation of NLQ on the top of the semantic layer), etc.
We're tackling this with a pretty different angle. Stateful data store managing intelligent connectors. This means you can connect to 3+ business systems, manage data quality/accuracy within those systems (statefully), then produce on-demand joins for a shared metrics library with the new Insights product.
So we vote someone else wins ;) what else are startups for but to chomp at the heels of behemoths? And what else are behemoths-at-war for but to leave an eventually empty battlefield to be painlessly overtaken by a new player?
An interesting topic that, to me, is nothing new. As a BI engineer something I have worked since I started. The only difference is that it is now ripped out of the BI tool and added additional features. Interesting to see the Evolution of the Semantic Layer 📈 :
1991: SAP BusinessObjects Universe and BI semantic layer
2008: Master Data Management (MDM) (with MDS from Microsoft in 2008)
2013: Kimball discussed the concept of a semantic layer in #158 Making Sense of the Semantic Layer
2016: Maturing BI tools with an integrated semantic layer such as Tableau, TARGIT, PowerBI, Apache Superset, etc. have their own metrics layer definition
2018: Jinja templates and dbt eroding the transformation layer into a semantic layer
2019: Looker and LookML popularized as the first real semantic layer
2022: Modern Semantic Layer, Metric Layer or Headless BI tools such as MetriQL, MetricFlow, Minerva, dbt arose with the explosion of data tools (BI tools, notebooks, spreadsheets, machine learning models, data apps, reverse ETL, …)
More on https://airbyte.com/blog/the-rise-of-the-semantic-layer-metrics-on-the-fly in case of interest.
https://www.kimballgroup.com/2013/08/design-tip-158-making-sense-of-the-semantic-layer/
dbt Metrics are so immature. I think that the definition of metric by dbt shows the constrains their approach has...
dbt Metrics definition:
A metric is a timeseries aggregation over a table that supports zero or more dimensions.
The need for time dimension and aggregation over a (single) table is very limiting.
I recommend checking the following article: https://medium.com/gooddata-developers/gooddata-and-dbt-metrics-aa8edd3da4e3
The article compares GoodData metrics with dbt metrics, and it is really interesting to see the key differences.
Thanks for sharing that - it was a nice read.
Agreed dbt's metrics are quite limiting, but if you squint your eyes, there isn't really much stopping dbt from adding in foreign key relationships and removing a lot of those limits. There's some added friction in that it takes a lot more boilerplate to get a metric down, but this does have its benefits in giving more weight to the metric itself (over the rest of the semantic layer components).
At the end of the day, my big question is: will the power of dbt/looker's integrations/momentum win out? Or will something with a different (maybe better) user experience win (like GoodData, but to be fair, cube or malloy or some other startup)? Is this a game of integrations/momentum or a game of user experience? At this point, I'm not sure, but it's generally really hard to bet against momentum
I think that the most crucial thing for metrics together with the semantic layer is standardisation. It would be so much easier if there was a standardised open-source semantic layer that would be utilised and extended by other vendors (dbt, Looker, GoodData, cube, etc.). It would solve many things, such as vendor lock, features portability (implementation of NLQ on the top of the semantic layer), etc.
"I can’t be sure but iirc dbt’s syntax looks really suspiciously like Supergrain’s did before they pivoted."
LOL pull George in here ;)
Ha yeah definitely need to get his take one of these days
Might look at Microsoft’s “metric layer” in their MS Fabric semantic model. Been around about 14 years now.
lol "something something data mesh"
We're tackling this with a pretty different angle. Stateful data store managing intelligent connectors. This means you can connect to 3+ business systems, manage data quality/accuracy within those systems (statefully), then produce on-demand joins for a shared metrics library with the new Insights product.
http://syncari.com/insights/
https://syncari.com/product/
So we vote someone else wins ;) what else are startups for but to chomp at the heels of behemoths? And what else are behemoths-at-war for but to leave an eventually empty battlefield to be painlessly overtaken by a new player?