14 Sep Version 0.9.0 release
New dbtvault version release
(v0.9.0) – Long awaited update
It has been a few months since our last release (v0.8.2/v0.8.3) – which was released earlier this year. Version 0.8.x was a very big update to our dbtvault package introducing Google BigQuery and MS SQL server support along with various bug fixes.
Our engineers have been working hard to bring this new version of dbtvault out to the public and today is the day – v0.9.0 includes the following exciting updates:
· Initial Postgres and Databricks support (Hubs, Links, Satellite support)
· Built-in Business Key Null Value handling
· Payload column exclusion
· A new Extra columns Parameter
· Various important bug fixes
Let’s have a look at these new features in more detail:
Postgres and Databricks (Hubs, Links, Satellite support)
We are delighted to release the first wave of Postgres and Databricks support. This is of course, monumental. The community has been asking for these for a long time. The development effort to support these platforms has also been contributed towards by the community, which we are incredibly grateful for. This collaboration was done behind the scenes due to complexities using the dbtvault development environment, but we will soon be releasing public contribution guidelines using our learning and experience from this collaboration over the last few months!
Payload column exclusion
Users have been able to define a Hashdiff based on the columns they do not want in the Hashdiff, instead of the columns they do. This is very useful for large Hashdiffs, where listing every column manually is very time-consuming. Now, users can do the same with payloads!
Long awaited: Built-in Business Key Null Value handling
This feature allows users to configure what happens when data arrives empty in the system and impose business requirements with simple configurations in the stage layer. To do this before, users would have had to create business rules or create lots of intermediate table/view layers to process NULL values work around the issue. This is a considerable quality of life improvement. In addition, we are currently working on developing built-in Ghost Key functionality for a future release.
Extra columns Parameter
A new parameter (src_extra_columns) now allows users to provide arbitrary columns to tablemacros. This allows for a lot more flexibility and caters for business-specific use cases where columns other than the standard columns, are needed in the resulting Hub, Link, Satellite etc. This includes columns such as those to handle multi-tenancy, and more
Bug Fixes
· Fixed an edge case for vault_insert_by_period when the staging table and the target table were in different databases
· Removed the uppercase conversion in the staging macro
· Fixed an issue where duplicate records (same hashdiff) would sometimes be loaded into a Satellite
· Disabled automatic column name escaping in derived columns when using the stage() macro. Escaping can now be configured on a case-by-case basis for each column to escape when they are reserved words etc.
You can read more about the release on our GitHub releases page.
We are very proud of the automate-dv.community and would like to thank you for your contributions and suggestions which allow us to improve our tool! dbtvault is build for the community (and now by the community) and we would love to hear your feedback – click here to share your thoughts.
Our work doesn’t stop here, we are continuing to work on further updates – if there is something we can introduce that will everyone’s experience with dbtvault more efficient and better, please get in touch with us.