databricks

FEBRUARY 27, 2024

Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the S.

Data Engineering 272

Data Engineering Data Engineer Data Engineering Data Engineering 272

databricks

FEBRUARY 28, 2024

This post is the second part of our two-part series on the latest performance improvements of stateful pipelines. The first part of this.

Data Engineering 231

Data Engineering Data Engineer Data Engineering Data Engineering 231

Towards AI

FEBRUARY 12, 2024

Photo by Samuel Sianipar on Unsplash Originally published on my blog. It’s not uncommon to be caught up in long debugging cycles when working with Spark. I was recently caught in such a debugging train when one of my pipelines was taking longer than expected. When processing big data, efficiency is key.

Big Data 82

Big Data Big Data AI AI 82

The MLOps Blog

JANUARY 26, 2024

To generate value from your model, it should make many predictions, and these predictions should improve a product or lead to better decisions. In this article, I’ll introduce you to a unified architecture for ML systems built around the idea of FTI pipelines and a feature store as the central component. But what is an ML pipeline?

Machine Learning 73

Machine Learning Machine Learning ML ML 73