01 Nov Coalesce

Datavault and the dbtvault Team Attend Coalesce 2022, London

It was well worth the early morning starts for all the Datavault team attending the Coalesce 2022 London (Online+) Analytics Engineering Conference. Attending were the four internal contributors for dbtvault: Alex Higgs, Aakash Datta, Jocelyn Shannon and Tim Wilson.

The team’s experience in dbt and the industry was wide ranging, with two senior engineer’s (Tim and Alex) and two of the newest Datavault starters, Jocelyn and Aakash.

Alex has been involved in dbt meetups for the last two years, but for everyone else this was their first in-person dbt event. It’s fair to say that there was something for everyone, however

The team had high praise for the conference:

We had a fantastic day at the Coalesce 2022 conference. We can’t explain how welcoming and friendly an atmosphere it was. It’s such a great community to be part of and it really felt like a family. There were lots of interesting discussions for new and experienced users of dbt alike. We’re already looking forward to next year’s conference, and the new and exciting features announced at the conference which are in store for dbt users everywhere!

What new features you ask? The dbt Labs team did not hold back.

1. The dbt Semantic Layer This is the big one. First off, a bit of a history lesson! dbt was born with the idea to bring software development best practises and development approaches to data warehousing and engineering: documentation, testing, DRY principles and more. Today, dbt provides analytics engineers and other data engineering professionals with the means to create their data products with a standardised, streamlined and carefully curated approach to development.

This is great for the developer, but what does it mean for the end user, the marketing teams, business analysts and other consumers of the data? It often turns into a very ‘hand-wavy’ explanation because all these standardised processes are in place, they should get more accurate data, faster.

This is all very well and, in most cases, true. In other cases, once the end-user or analysts gets their hands on the data and metrics, suddenly there’s multiple definitions for the same

metric, people are aggregating the data incorrectly and the data scientists are complaining they do not understand how the numbers were arrived at.

At this point some would say “Ah but that’s not dbt’s responsibility; the business has their own tools like PowerBI and Looker and the users are at fault for misusing their tools!”

Not anymore. Enter the dbt semantic Layer: bringing all the power of dbt (testing, documentation, and DRY principles) to metrics. This new dbt feature provides:

– The ability to define common definitions for your metrics using dbt.

– The means to integrate dbt into your favourite BI tools seamlessly

– A way forward for improving the way our end users access and create insights

The dbt Semantic Layer is essentially a Metrics API for dbt, which popular tools will now be able to take advantage of to integrate into dbt. This the beginnings of an open standard for metrics in the industry.

Some tools have had a head start, such as Lightdash, an open-source dbt-integrated Dashboarding tool (Read their blog post on this!)

Keep in mind this is cutting-edge, brand-new functionality. It’s not yet matured into what it could be, but if this is the beginning then I am personally itching to see how this feature evolves!

To quote Drew Banin, Co-founder of dbt Lab, at the conference in London today:

“The dbt semantic layer brings semantics closer to the data”

And that can only be good for the industry and dbt users everywhere.

Further reading: – Frontiers of the Semantic Layer: Extending the dbt Workflow (getdbt.com)

2. Python + dbt in perfect harmony: Python dbt models
Python is the go-to language for the data world (other than SQL, of course) and it comes as no surprise that dbt has set its sights on a closer relationship with the do-it-all data developer’s language of choice, releasing with dbt 1.3.

This functionality is still in its infancy and as a community, we do not yet know the limits of what it can do. This shouldn’t be a worry or a deterrent however: this should be considered exciting, and the unknown should be enticing, in the same way that I imagine stepping foot on a new planet might be.

I for one welcome this new addition to dbt, as someone who has developed Python integrations and wrappers for dbt extensively, particularly for dbtvault’s test suite.

The possibilities here are potentially endless, and with discussions ongoing it’s an exciting time to be developing with dbt.

Further reading:

– Polyglot pipelines: why dbt and Python were always meant to be (getdbt.com)

– Introducing support for Python, dbt’s second language (getdbt.com)

3. A new look and feel for the dbt Cloud IDE Long time users of dbt Cloud will have a very love-hate relationship with it. When it works, it works like a charm and is a trusty companion to dbt developers everywhere, when it doesn’t work well, you may wish you were back in the CLI (I’m looking at you Tim!)

dbt has known about these issues for a while and has been working very hard developing new and improved functionality behind the scenes!

Major improvements include:

– Performance

– Autocomplete

– Code Formatter

– Diff View

– UI Organisation and Navigation

Announcing the New and Improved Cloud IDE (getdbt.com)

Other than the announcement and discussion of all these wonderful and exciting new features, there were great presentations from many different companies and users of dbt.

During the “The Big Ideas – Hot Takes on Hot Takes” panel, Data Mesh was a hot topic, as well as Data Contracts and Open Data formats, all fantastic topics that were great to listen to the panel discuss.

Slack was used as a platform to ask questions or discuss the current topic amongst the crowd during presentations. I posted the following during the discussion on Data Contracts:

This was called out by Nick Spitzer, Sales Director of dbt Labs as a great point; Drew Banin (dbt Labs Co-Founder) and Julia Schottenstein, product manager at dbt Labs, agreed!

I personally hugely appreciated this level of audience interaction with some of the biggest names at dbt Labs.

Thanks for another great conference dbt Labs, see you next year!

Alex Higgs