Flipboard

NOVEMBER 24, 2023

Iris was designed to use machine learning (ML) algorithms to predict the next steps in building a data pipeline. Let’s combine these suggestions to improve upon our original prompt: Human: Your job is to act as an expert on ETL pipelines.

Database 159

Database AWS ETL SQL 159

Dataconomy

MAY 16, 2025

A 2025 landscape analysis shows ApacheKafka , Flink , and Iceberg moving from niche tools to fundamental parts of modern data architecture, underscoring how ubiquitous realtime expectations have become. Common pitfalls and how to avoid them Tomlein highlights five recurring traps: Data leakage Partition feature calcs strictly by event time.

AI 103

AI AI ETL ML 103

The MLOps Blog

SEPTEMBER 7, 2023

Data Scientists and ML Engineers typically write lots and lots of code. From writing code for doing exploratory analysis, experimentation code for modeling, ETLs for creating training datasets, Airflow (or similar) code to generate DAGs, REST APIs, streaming jobs, monitoring jobs, etc. Related post MLOps Is an Extension of DevOps.

Machine Learning 52

Machine Learning Machine Learning ETL ML 52

phData

MARCH 1, 2024

This functionality eliminates the need for manual schema adjustments, streamlining the data ingestion process and ensuring quicker access to data for their consumers. As you can see in the above demo, it is incredibly simple to use INFER_SCHEMA and SCHEMA EVOLUTION features to speed up data ingestion into Snowflake.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

phData

AUGUST 22, 2024

Consider a data pipeline that detects its own failures, diagnoses the issue, and recommends the fix—all automatically. This is the potential of self-healing pipelines, and this blog explores how to implement them using dbt, Snowflake Cortex , and GitHub Actions. This output is less helpful.

SQL 52

SQL Data Quality Python Data Warehouse 52

phData

JUNE 7, 2023

The best part of this step is that focusing on building a strong data foundation and operational maturity around data pipelines will not only help prepare you for AI success but is also a critical step for more traditional analytics maturity and becoming a more data-driven organization.

AI 59

AI AI Machine Learning Machine Learning 59

Alation

APRIL 14, 2023

Last week, the Alation team had the privilege of joining IT professionals, business leaders, and data analysts and scientists for the Modern Data Stack Conference in San Francisco. We had a great time meeting with customers and demonstrating how a data intelligence platform delivers visibility across the data stack with demos.

Data Pipeline 52

Data Pipeline Data Warehouse Cloud Data ETL 52