17 Nov Overcoming Complexity with dbt at Coalesce London 2023
Overcoming Complexity with dbt at Coalesce London 2023
At dbt Labs’ Coalesce London 2023, Alex Higgs, product manager of AutomateDV and senior consultant data engineer at Datavault, presented “Demystifying Data Vault with dbt” to a packed room of data enthusiasts and business professionals.
Alex addressed the audience’s concerns head-on, unravelling the mysteries of Data Vault and how to use AutomateDV and dbt (data build tool) to automate the process.
Understanding the Audience and Their Challenges
Alex kicked off the session by identifying the audience’s challenges – all of which he has come across as a data engineer at Datavault. The key areas of concern included questions like:
- What is Data Vault?
- Is Data Vault worth the investment?
- How can I start a Data Vault project?
- Strategies for scaling data solutions.
- Methods to reduce development costs.
The presentation answered all these questions, giving the audience actionable tips to solve their data problems.
Sammy the Cat: An Analogy for Data Vault Newbies
To simplify the complex concept of Data Vault for newcomers, Alex told the story of when he first got his cat, Sammy.
Sammy was a stray rescue cat who used to hide in the corner of Alex’s home for hours when he first arrived in the new environment.
Much like Sammy’s initial hesitance in his new environment, it’s common for people new to Data Vault to be apprehensive and unsure of where to begin. Much like how Alex nurtured the relationship and provided a safe environment for Sammy, Alex has worked hard to simplify the complexities of Data Vault using dbt by developing AutomateDV.
What is Data Vault? A Brief History and Essential Components
Dan Linstedt developed Data Vault 1.0 in 2000 as a response to the overwhelming challenge of handling vast amounts of data in the US Department of Defence.
Fast forward to 2013, Data Vault 2.0 emerged, building on and further developing the principles of scalability, agility, integration, real-time feeds, auditability, automation, and flexibility that were pioneered in the original methodology.
Read Datavault’s blog, ‘What is Data Vault?’ to learn more about the methodology.
When is the right time for Data Vault?
Data Vault designed to be a a silver bullet, however we see the best results business’s need…
…to be agile
…integration of multiple systems
…real-time feeds
…scalability
…auditability
…automation
…flexibility
Read Datavault’s blog, ‘When is the right time to use Data Vault?’ to learn more.
Common mistakes and how to overcome them
Alex shed light on some common mistakes he has come across in the field, consulting with a variety of different business implementing or considering Data Vault. These included:
Mistake: Modelling from source system data rather than the business point of view (i.e., creating a source system data vault).
Solution: Spend more time with the business, understand their needs. Carry out data modelling!
Mistake: Thinking “we don’t need to do data modelling”.
Solution: Yes, you do. You need to make the data work for the business, and deliver business value, otherwise why build a data warehouse?
Mistake: Just reading the book and starting a project.
Solution: Good start, but you should seek advice from an expert as well.
Mistake: Launching a project without Data Vault expertise and working to deadlines whilst learning a new approach to Data Warehousing.
Solution: Start with a focused proof-of-concept for a specific problem and seek review from experienced Data Vault practitioners.
Mistake: Not demonstrating business value from Data Vault.
Solution: Scope is key: start small and build incremental value along the way.
Alex advocated a meticulous approach, involving expert consultation, focused proof-of-concepts, and incremental value delivery.
Data Vault and dbt: A Perfect Integration
A highlight of Alex’s presentation was the mention of the seamless integration of Data Vault with dbt using AutomateDV.
dbt’s power and AutomateDV’s macros help users generate Data Vault structures as templates. The perfect match of Data Vault and dbt means AutomateDV is in the top 20% of most installed packages on dbt Hub.
But don’t just take our word for it. See how Oodle Car Finance leveraged AutomateDV to deliver exceptional company-wide results.
How to get started with Data Vault
AutomateDV offers a free way to get started with Data Vault, helping to learn the method a facilitating experimentation and proof-of-concepts.
There are also many other resources available to start your Data Vault journey:
Data Vault Q&A Forum – a free Q&A forum run by the Data Vault User Group, filled with industry experts like Patrick Cuba and Neil Strange ready to answer any questions you may have about Data Vault.
Data Vault: Core Concepts – a free, 2-hour virtual training session providing you with the fundamentals of Data Vault
Certified Data Vault Practitioner Course (CDVP2) – The official paid course to become a certified Data Vault practitioner.
The Future: A Scalable and Pattern-Based Solution
As the presentation ended, Alex painted a visionary picture of the future. With Data Vault and dbt at the helm, the future of data management appears scalable, pattern-based, and, most importantly, automatable. Alex Higgs’ presentation at dbt Coalesce London 2023 guided attendees through the complex world of Data Vault.