Applied Data Science

MAY 6, 2021

To keep myself sane, I use Airflow to automate tasks with simple, reusable pieces of code for frequently repeated elements of projects, for example: Web scraping ETL Database management Feature building and data validation And much more! What’s Airflow, and why’s it so good? What makes it my go to?

Data Scientist 130

Data Scientist Python Data Science Database 130

Data Science Blog

JUNE 27, 2023

Kappa – Architecture Jay Kreps introduced the Kappa architecture in 2014 as an alternative to the Lambda architecture. It offers the advantage of having a single ETL platform to develop and maintain. It is well-suited for developing data systems that emphasize online learning and do not require a separate batch layer.

Big Data 130

Big Data Big Data Apache Kafka Database 130

phData

NOVEMBER 9, 2023

Effectively this is a way to store the source of truth and build (or rebuild) your downstream data products (including data warehouses) from it. What is the Difference Between a Data Lake and a Data Warehouse? Historically, there were big differences.

Data Warehouse 98

Data Warehouse Data Lakes Clustering Cloud Data 98

DagsHub

OCTOBER 23, 2024

General Purpose Tools These tools help manage the unstructured data pipeline to varying degrees, with some encompassing data collection, storage, processing, analysis, and visualization. DagsHub's Data Engine DagsHub's Data Engine is a centralized platform for teams to manage and use their datasets effectively.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52