Interesting article. Who is going to win the EL space in long run ? I don't see snowflake or even Google has any great integration tools for data ingestion..
I am exploring ELT tools and no one seems scalable enough to handle my 10000+ sources.
Technically there are 2 distinct sources like SQL server, Sybase but I have to load data from 10K+ customers where data resides in on premises databases.
Technically there are 2 distinct sources like SQL server, Sybase but I have to load data from 10K+ customers where data resides in on premises databases.
write one DAG for each database and then loop through the different customers, you are only limited by how many workers your Airflow server can handle simultaneously
Teradata has less than half the customers they did at their height, and if you investigate where most of those customers went, I think you'll find that most of them went to Snowflake. The fact is, Teradata was very late to the Cloud, fumbled badly when they finally tried, and Vantage hasn't really stemmed the tide. Zero chance of Teradata being able to acquire dbt.
Bruhhh you clickbaited me!!!
I'm actually just our Chief Clickbait Officer, Seth. The truth finally comes out.
Interesting article. Who is going to win the EL space in long run ? I don't see snowflake or even Google has any great integration tools for data ingestion..
I am exploring ELT tools and no one seems scalable enough to handle my 10000+ sources.
Great point. The question of who's going to win the upstream battle is an interesting on.
Also, what the heck 10000+ SOURCES? What's your industry? :)
Technically there are 2 distinct sources like SQL server, Sybase but I have to load data from 10K+ customers where data resides in on premises databases.
If you have that many sources hire a Data Engineer and run your own ELT tool in airflow
Technically there are 2 distinct sources like SQL server, Sybase but I have to load data from 10K+ customers where data resides in on premises databases.
answer is still Airflow
write one DAG for each database and then loop through the different customers, you are only limited by how many workers your Airflow server can handle simultaneously
Look at qlik and fivetran. Both can help you to send this sources to Snowflake that can manage massive data
Teradata is way better than Snowflake, they should be the ones that will be acquiring dbt
Teradata has less than half the customers they did at their height, and if you investigate where most of those customers went, I think you'll find that most of them went to Snowflake. The fact is, Teradata was very late to the Cloud, fumbled badly when they finally tried, and Vantage hasn't really stemmed the tide. Zero chance of Teradata being able to acquire dbt.
you can't imagine the horror I went through when it popped on my feed. I am about to invest in a tool stack involving DBT. worst clickbait ever