Welcome to our new AutomateDV website and social media channels. You can follow us on your favourite platforms:
What is AutomateDV?
AutomateDV is a free, open-source tool for building Data Vault data platforms/warehouses.
In strict terms, it is a ‘package’ that runs on the dbt™ data engineering product. The AutomateDV package doesn’t constrain how you use dbt™ – it can be used to build other parts of the data pipeline that don’t need Data Vault patterns.
AutomateDV uses metadata to generate and execute the ETL code needed for a Data Vault warehouse, removing the need to write SQL. This leads to better quality code, fewer mistakes and improved productivity resulting in increased agility.
Does AutomateDV perform at enterprise scale?
AutomateDV is operating at scale in a large US insurer working with billion row tables where they were implementing Data Vault for the first time. They modelled difficult, complex data sets processing tables up to 54 terabytes in size.
Another telco user in the UK has operated with tables in excess of a billion rows without any performance difficulties.
Is AutomateDV suitable for prototyping and learning about the Data Vault method?
Many of the early adopters of AutomateDV were using it as an easy, low cost / low risk way of experimenting with the Data Vault method. We had spoken to many data engineers who said “we like what we hear about Data Vault but we have not used it before and we would like to be able to show that it would work in our organisation”. They used it to prototype and run proof of concept exercises to convince their management of the benefits of using the Data Vault method.
Why use AutomateDV?
There are many reasons to use AutomateDV. Here are a few!
- A Data Vault generating tool taking full advantage of Data Vault 2.0™ and its standardised structures which facilitate automation
- It is free and open source giving an easy-entry method to learn and use Data Vault
- It has been used in many large and small projects
- It offers time and manpower savings eliminating the need to write complex SQL.
- Proof of Concept: Prove the Data Vault 2.0™ method before investing in expensive tool alternatives
What is dbt™ (data build tool)?
dbt™ is a great open-source tool from dbt™ Labs. In their words it “is a command line tool that enables data analysts and engineers to transform data in their warehouses more effectively”. It is designed to be the “T” in ELT/ETL
What databases are supported?
We believe data warehouses will increasingly be cloud-based and we chose Snowflake for the initial implementation, as a highly performant, scalable database specifically designed to operate in the cloud.
AutomateDV supports a growing range of database platforms including Snowflake, Google BigQuery and Microsoft SQL Server. Check the latest here because other databases will be supported in due course.
Why does Data Vault lend itself to this type of automation?
Data Vault lends itself to automation because it is pattern based, meaning that SQL can be written for each pattern and parameterised to run each time the pattern is encountered in the production system. The use of automation dramatically improves productivity, maintainability and makes for faster, more reliable implementations.
How do I contact AutomateDV’s creators?
We are building a community. Join our Slack channel to participate. We would love to hear about your experiences and would welcome your contributions and suggestions email us on info@automate-dv.com.
Where do I go for AutomateDV consultancy?
We offer a wide range of consultancy services across different time zones including:
- Get new features developed by AutomateDV’s creators
- Get new functionality added to dbtvault
- Turnkey project delivery
- Deliver your projects with low-risk, agile implementation
- Avoid those early mistakes by using our coaching services to get you started
- Avoid those early mistakes: use our coaching service to get you started on your Data Vault and AutomateDV journey
- Team augmentation
- Don’t be limited by your resource availability, scale-up your team and accelerate your project
- Pilots and prototypes
- Keep your team focused on supporting new business needs: use us to maintain data pipelines, develop analytics, and administer/monitor the platform
- Training
- Talk to us about your training needs.
Contact us to discuss your needs info@automate-dv.com.
Which of the Data Vault table types does AutomateDV support?
AutomateDV supports and extensive range of different Data Vault structures including many of the more unusual ones!
- Hubs
- Links
- Transactional Links
- Satellites
- Effectivity Satellites
- Multi-Active Satellites
- Extended Tracking Satellites
- As of Date Tables
- Point In Time (PIT) tables
- Bridge Tables
Is AutomateDV proven?
AutomateDV has been used by many organisations large and small. Hear from some of those that have talked about their experiences including venture capital company Georgian Partners in Canada and property company Mirvac in Australia.
How do I get AutomateDV support?
Users of AutomateDV have the reassurance of the AutomateDV support services which allows users to confidently deploy AutomateDV in their organisations with on-going assistance from the creators of the product.
The service is designed to help diagnose your problems and resolve them quickly with access to senior developers who understand data warehousing, Data Vault and dbt™ as well as AutomateDV.
Find out more here.
What is the history of AutomateDV?
Originally, we developed a Data Vault(tm) generator tool based on dbt(tm) for in-house purposes. We quickly realised our tool would be useful to others, and so dbtvault was born and released on an open-source basis.
Since release, we decided to rename the product to Automate Data Vault (AutomateDV) to showcase the purpose of the tool. We have gradually built out functionality – we welcome suggestions or contributions from the community.
What technology does AutomateDV use?
AutomateDV uses an open-source tool written in Python called dbt™. AutomateDV is a dbt™ package, i.e.it runs within dbt™ and dbt™ has a wider application than just Data Vault data warehouses.
Why do we like dbt™?
We like dbt™ because it is designed in line with best practices such as “Don’t Repeat Yourself” and has ease of maintenance in mind. It makes strong use of templates and macros and comes with many useful features such as:
- Clean transformations
- dbt™ allows you to standardise your data transformations using models
- Reduce errors
- By minimising the amount of handwritten SQL, it limits the number of errors
- Performance
- Multi-thread and optional incremental loading enable effective performance management
- Auditability
- There are audit and diagnosis tools with full logging, and it generates lineage graphs
- Pipeline building
- You can develop your data pipeline from raw data to BI tools
- Active community
- There is an active and accessible community to contribute and share ideas
- Extensibility
- We have used the extensibility features of dbt™ to create our AutomateDV package.
What are the features of AutomateDV?
AutomateDV is specifically designed to help you automate the loading of a Data Vault data warehouse. It is easy to use and install and use for a competent Data Engineer.
- It is metadata driven meaning that there is little or no SQL required.
- The software is fully tested using Behavioural and Unit tests
- Comprehensive documentation is available on ReadTheDocs
- There is a worked example using Snowflake supplied data for hands-on learning.
- The template supports one of the most extensive ranges of Data Vault structures: hub, link, satellite, transactional links, and many more.
How do I get started with AutomateDV?
To find out more about using AutomateDV go to:
- Learn about Data Vault from the Datavault “What is Data Vault“ page.
- Get the software on GitHub.
- Read the documentation on Readthedocs.
- Check-out our worked example in our documentation for hands-on learning.
- Sign-up for our Slack channel for updates and discussion on the product.
- Follow us on social media for news and updates.
- Watch videos on our YouTube channel.
Who owns AutomateDV?
AutomateDV was developed by the specialist data platform consultancy Datavault and is available on an open-source basis via GitHub.
Is there an AutomateDV tutorial?
There is extensive AutomateDV documentation on Readthedocs and we have incorporated a worked example into our documentation that provides hands-on learning.
Additionally, we are working on a AutomateDV training course that should be available in the coming months.