21 Jun How does AutomateDV and dbt Cloud support Data Vault?
How does AutomateDV and dbt Cloud support Data Vault?
Demystifying Data Vault
To mark the rebrand of dbtvault to AutomateDV, we hosted a webinar with dbt labs. Alex Higgs, lead developer of AutomateDV, and Jeremy Cohen, product manager at dbt labs, explained the benefits of using the Data Vault method when building corporate data platforms. They then went on to explain how data engineering teams will get value from AutomateDV and dbt Cloud by making Data Vault easily accessible.
In this blog, we’ll break down these topics from the webinar into bitesize nuggets. If you want more detail, the full recording is now available on our YouTube channel.
The topics we’ll cover include:
- What is Data Vault?
- Why use Data Vault?
- When to use Data Vault
- What is AutomateDV?
- How does AutomateDV help with Data Vault?
- How does dbt support Data Vault?
What is Data Vault?
Dan Linstedt, the inventor of the method, defined Data Vault as “a system of business intelligence that provides the necessary components required to accomplish enterprise vision in data warehousing and information delivery”.
As it is a “system of business intelligence”, that means that it is not only data modelling, but also architecture, SQL patterns, people, and process (ways of working). Likewise, the statement “enterprise vision” suggests that Data Vault takes a business view across the enterprise – meaning it works best when combining multiple data sources into a combined business view.
Why use Data Vault?
Data Vault provides you with an opportunity to build a data platform that meets your business’s needs. When starting out on your Data Vault journey, prioritise the delivery of business value and avoid trying to “boil the ocean”. Data Vault allows you to incrementally build to a scale that suits your business.
We’ve put together 8 of the top reasons your business should be using Data Vault to realise the value of your data:
- Ease of integration
- Auditability
- Owned by the business
- Scalability
- Adaptability
- Automation-driven productivity
- Agile ways of working
- Enterprise-wide coverage
For a detailed look at why you should use Data Vault, read a blog on this topic by Datavault here.
When to use Data Vault
Data Vault could be a good option for you when you need to be…
- Agile
- Real-time
- Auditable
- Flexible
- Integrated in multiple systems
- Scaled to an enterprise level
- Automated
Data Vault isn’t for every organisation. It’s important to consider this list when you are deciding whether you are going to use it or not.
What is AutomateDV?
AutomateDV makes building a Data Vault so much easier. Our team created it in 2019 as an open-source Data Vault automation tool to give back to the community. Originally named dbtvault, here’s how our tool can help you:
- Multiple diverse data sources – load data from a variety of sources.
- Metadata driven – you provide only metadata to AutomateDV.
- Generate your Data Vault – no SQL coding required when you use AutomateDV (structures supported are Hubs, Links, Satellites, effectivity satellites, multi-active satellites, extended tracking satellites, as of date tables, point-in-time tables, and bridge tables).
- dbt transformation – Built on dbt for rapid and trusted development workflows.
- Data platforms – AutomateDV supports a range of data platforms including BigQuery, PostGres, Databricks, and more.
- Deliver insight – analytics, artificial intelligence, and machine learning.
How AutomateDV helps with Data Vault
AutomateDV simplifies Data Vault with its unique capabilities using the dbt package.
AutomateDV and dbt operate in the “transform” and “load” sections of your data platform. In strict terms, it is a package that runs on the dbt. AutomateDV doesn’t constrain how you use dbt. You still benefit from all the features available from dbt, but with our added extra components necessary to building a Data Vault.
How does dbt support Data Vault?
dbt has created “a hosted service that helps data analysts and engineers productionise dbt deployments.”. dbt Cloud offers features like job scheduling, CI/CD, documentation serving, and monitoring.
The introduction of dbt cloud simplifies job configuration, and the run History interface facilitates job tracking. The new release also includes a tighter API integration and a unified look and feel for improved user experience.
Summary
In summary, we’re excited for the future of AutomateDV and dbt Cloud to help enterprises deliver extraordinary business value.
AutomateDV is one of the top 20 most installed packages (on the dbt marketplace), with hundreds of projects actively using it in production.
A notable comment from Jeremy Cohen in the webinar was when he said that AutomateDV is “one of the top 20 most installed packages (on dbt), with hundreds of projects actively using it in production.”. This is further evidence that AutomateDV is serving the community at scale, with the support of dbt.