EL Tool + dbt - T Tool

Fivetran is another cloud native tool that we like to work with. It is particularly good when we have a lot of data objects to bring into the data warehouse or operational data store. This is because it provides a very low labor (low engineering cost) way to bring in data. Like Matillion it has many off the shelf integrations for connecting to data sources and is very good at extracting and loading the data. However it does not provide any useful features for transforming the data once it is in our DW or ODS. To that end we use a tool called dbt which stands for “data build tool”. This tool has gained a lot of popularity of late. We call it “sql on steroids” as it is a python based tool that uses sql with a ginga templating language and then “compiles” the code to sql only. Pros: Low labor to get a lot of data into the raw layer of the data warehouse. Cons: FiveTran costs can get expensive for very large volumes of data.