19 Apr Build for scale, agility, and reliability with dbt Cloud, AutomateDV and Data Vault
Build for scale, agility, and reliability with dbt Cloud, AutomateDV and Data Vault
On the 19th of March 2024, we collaborated with dbt Labs in an insightful webinar titled “Building a Warehouse with Scale, Agility, and Reliability.” The session was led by Sean McIntyre, a Solutions Architect at dbt Labs, and Alex Higgs, a Senior Data Engineer at Datavault and Product Owner of AutomateDV.
Sean and Alex talked about how to use dbt Cloud and AutomateDV to build a Data Vault, helping over 200 attendees move away from inconsistent and siloed data models to a unified, scalable architecture that supports agile development and simplifies data complexity.
Let’s take a closer look at the content of the webinar. Alternatively, you can watch the full recording on our YouTube channel!
What is dbt?
dbt is the new standard for data transformation. In use at over 30,000 enterprises worldwide, and a thriving 100,000+ community, dbt has cemented its position as the go-to solution for modern data transformation needs.
But what exactly is dbt?
dbt is “a SQL-first transformation workflow that lets teams quickly and collaboratively deploy analytics code following software engineering best practices like modularity, portability, CI/CD, and documentation. Now anyone on the data team can safely contribute to production.”
At its core, dbt is a comprehensive platform that empowers businesses to extract actionable insights from their data swiftly and seamlessly. Once data is loaded into the cloud platform (such as Snowflake), dbt comes into play, facilitating the generation of insights through dashboards, integration with other platforms, or even artificial intelligence applications.
One of dbt’s key strengths lies in its ability to provide consistency, thereby assisting businesses in achieving their goals, whether it’s boosting revenue, meeting project deadlines, or enhancing overall operational efficiency. dbt Cloud, the cloud-based offering, extends beyond the capabilities of dbt Core, serving as the standard for data transformation at scale.
The dbt Cloud Flywheel: Driving Success
dbt Cloud is the standard for data transformation at scale. Central to dbt’s philosophy is the establishment of a symbiotic relationship between developers and consumers. This dynamic synergy propels the data velocity forward, enabling organizations to develop, test, and deploy data in a standardized manner.
Moreover, dbt’s emphasis on data quality translates into improved trust, ultimately fuelling greater demand for data-driven insights. By optimising costs and ensuring better data return on investment, dbt empowers businesses to maximise the value derived from their data assets.
What is AutomateDV?
AutomateDV is the Data Vault automation tool powered by dbt. It is available as an open-source tool on dbt’s product marketplace and enables data engineers to get started with Data Vault quickly and efficiently.
Combining the prowess of dbt with advanced macros, AutomateDV eases Data Vault generation. Automation is achieved through pattern-based SQL generation simplifying the process of creating and managing Data Vault structures, enabling organisations to effortlessly and efficiently deploy Data Vault solutions.
The Power of Data Vault: Meeting Business Needs
At its core, the Data Vault method aligns closely with the fundamental requirements of businesses by:
Identification and definition of business objects: Data Vault enables organisations to identify and define key business objects, laying the foundation for meaningful insights.
Understanding business object interactions: By capturing the relationships and interactions between business objects, Data Vault provides a comprehensive view of organizational dynamics.
Tracking business object state over time: With Data Vault, organisations can track the state of business objects and their interactions, facilitating historical analysis and trend identification.
The Anatomy of an AutomateDV Project
AutomateDV projects embody efficiency and agility, offering a glimpse into the future of data management. With a focus on pattern-based SQL generation using the features of dbt Cloud, AutomateDV projects exemplify key components of the modern data stack.
McDonald’s Nordics used AutomateDV and dbt Cloud to establish a nimble and standardised Data Vault structure. By leveraging macros and the AutomateDV package, McDonald’s Nordics unifies disparate data stacks across franchise locations and tracks $1.4bn in sales across restaurants, drive-throughs, and deliveries in four markets.
Why Build a Data Vault with dbt?
When striving for effective data management, organisations often explore various approaches, such as stored procedures or drag-and-drop tools. However, these methods can prove to be inadequate in the long run, leading to dissatisfaction among data consumers and confusion regarding data lineage and governance.
dbt brings modularity, version control, and testing to the table, supporting you in the process of building a Data Vault. With dbt Cloud, organisations can not only build and scale their Data Vaults but also iterate and deploy changes rapidly, all while maintaining elevated standards of quality and governance.
dbt Cloud: Enabling the Data Lifecycle
dbt Cloud serves as the cornerstone of the data lifecycle, offering a suite of features designed to streamline development, testing, deployment, and monitoring processes. From building and scaling data meshes to facilitating domain-level ownership of data, dbt Cloud empowers organizations to unleash the full potential of their data assets.
Leveraging dbt Cloud Features for Enhanced Data Governance
With built-in model contracts, version control, and lightweight data cataloguing capabilities, dbt Cloud ensures that organisations can navigate the complexities of data governance with ease. Additionally, through joint service offerings, such as rapid onboarding and migration support, dbt Cloud accelerates the journey towards data excellence.
Conclusion
In conclusion, our joint webinar with dbt Labs offered a comprehensive overview of the tools and methodologies shaping the future of data management. By embracing the power of dbt and AutomateDV, organisations can embark on a journey towards data-driven excellence, driving innovation, and unlocking new opportunities in the digital age.