Hacker News

JUNE 9, 2025

or book a demo: https://cal.com/shreyashn/chonkie-demo. reply snyy 5 hours ago | parent | next [–] As mentioned in the other reply, we have a cloud/on-prem offering that comes with a managed ETL pipeline built on top of our OSS offering. If you're interested, reach out at shreyash@chonkie.ai

Database 92

Database SQL ETL AI 92

Women in Big Data

MARCH 5, 2025

I worked extensively with ETL processes, PostgreSQL, and later, enterprise-scale data systems. Many companies struggle with data silos, so we focus on centralizing data, optimizing ETL processes, and enabling real-time analytics. Q: Do you have any book recommendations? Q: Tell me more about Data Surge?

ML 52

ML ML Big Data Big Data 52

Dataconomy

MAY 16, 2025

Teams needing subsecond decisions often push enriched events to Kafka or Kinesis via Snowbridge ; those consolidating on a warehouse can stream straight into Snowflake through the Snowplow Streaming Loader no duplicate ETL required. Trainingserving skew Source both phases from the same feature store. Ready to move from theory to throughput?

AI 103

AI AI ETL ML 103

ODSC - Open Data Science

JUNE 12, 2023

This article is an excerpt from the book Expert Data Modeling with Power BI, Third Edition by Soheil Bakhshi, a completely updated and revised edition of the bestselling guide to Power BI and data modeling. Then we have some other ETL processes to constantly land the past 5 years of data into the Datamarts. What is a Datamart?

Power BI 52

Power BI Data Warehouse ETL Data Preparation 52

IBM Journey to AI blog

APRIL 20, 2023

Read this e-book on building strong governance foundations Why automated data lineage is crucial for success Data lineage , the process of tracking the flow of data over time from origin to destination within a data pipeline, is essential to understand the full lifecycle of data and ensure regulatory compliance.

Database 86

Database Data Engineer Data Engineering Data Engineering 86

AWS Machine Learning Blog

DECEMBER 13, 2023

It’s useful for coordinating tasks, distributed processing, ETL (extract, transform, and load), and business process automation. Outside of work, he spends his time building things and watching comic book movies with his family. It handles the underlying complexity so you can focus on application logic.

AWS 127

AWS AI AI ETL 127

Alation

MAY 24, 2022

The Lineage & Dataflow API is a good example enabling customers to add ETL transformation logic to the lineage graph. In Alation, lineage provides added advantages of being able to add data flow objects, such as ETL transformations, perform impact analysis, and manually edit lineage. Book a demo today. The post Alation 2022.2:

Data Quality 98

Data Quality Data Governance ETL Data Observability 98

Dataversity

MARCH 1, 2021

Data warehouse disciplines and architectures are well established and often discussed in the press, books, and conferences. Data warehouse (DW) testers with data integration QA skills are in demand. They have become a standard necessity for most modern organizations. Each business often uses one or more data […].

Data Warehouse 52

Data Warehouse ETL Data Governance Data Quality 52

Analytics Vidhya

APRIL 19, 2021

ArticleVideo Book This article was published as a part of the Data Science Blogathon. Introduction ETL pipelines look different today than they used to. The post Is manual ETL better than No-Code ETL: Are ETL tools dead? appeared first on Analytics Vidhya.

ETL 343

ETL Data Science Analytics Analytics 343

ODSC - Open Data Science

APRIL 4, 2024

ODSC Highlights Announcing the Keynote and Featured Speakers for ODSC East 2024 The keynotes and featured speakers for ODSC East 2024 have won numerous awards, authored books and widely cited papers, and shaped the future of data science and AI with their research. Learn more about them here!

Data Visualization 52

Data Visualization Analytics Analytics Big Data Analytics 52

phData

NOVEMBER 9, 2023

Data Processing: Snowflake can process large datasets and perform data transformations, making it suitable for ETL (Extract, Transform, Load) processes. If you’d like a more personalized look into the potential of Snowflake for your business, definitely book one of our free Snowflake migration assessment sessions.

Data Warehouse 98

Data Warehouse Data Lakes Clustering Cloud Data 98

Heartbeat

NOVEMBER 6, 2023

You also learned how to build an Extract Transform Load (ETL) pipeline and discovered the automation capabilities of Apache Airflow for ETL pipelines. Image Source —  Pixel Production Inc In the previous article, you were introduced to the intricacies of data pipelines, including the two major types of existing data pipelines.

Data Pipeline 52

Data Pipeline Clean Data ETL Python 52

IBM Journey to AI blog

JULY 1, 2024

In her book, Data lineage from a business perspective , Dr. Irina Steenbeek introduces the concept of descriptive lineage as “a method to record metadata-based data lineage manually in a repository.” Critical and quick bridges The demand for lineage extends far beyond dedicated systems such as the ETL example.

ETL 40

ETL Data Lakes Database Data Pipeline 40

Alation

JUNE 14, 2023

You don’t have to write ETL jobs.” That lowers the barrier to entry because you don’t have to be an ETL developer. Register (and book a meeting with our team). Anyone building anything net-new publishes to Snowflake in a database driven by the use case and uses our commoditized web-based GUI ingestion framework.

Data Pipeline 52

Data Pipeline ETL Data Warehouse SQL 52

phData

JANUARY 19, 2024

For this project, we will utilize a simple OpenLibrary API to find many books based on a subject and a time window. Now, we’ll make a GET request to the following endpoint, which is set up to look for analytics books released between 2014 and 2024. Each API has its own set of requirements.

Data Pipeline 52

Data Pipeline Data Warehouse ETL Azure 52

phData

FEBRUARY 14, 2023

Tips When Considering Streamsets Data Collector: As a Snowflake partner, Streamsets includes very intricate documentation on using Data Collector with Snowflake, including this book you can read here. Data Collector can use Snowflake’s native Snowpipe in its pipelines.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

The MLOps Blog

AUGUST 3, 2023

At a high level, we are trying to make machine learning initiatives more human capital efficient by enabling teams to more easily get to production and maintain their model pipelines, ETLs, or workflows. We had books to make a model run in Spark or on a large box. How is DAGWorks different from other popular solutions?

ML 52

ML ML Data Scientist Machine Learning 52

phData

SEPTEMBER 27, 2024

If transitional modeling is like building with Legos, then activity schema modeling is like creating a flip book animation of your customer’s journey. In traditional ETL (Extract, Transform, Load) processes in CDPs, staging areas were often temporary holding pens for data. What is Activity Schema Modeling?

Data Models 52

Data Models Data Modeling Apache Kafka Data Lakes 52

How to Learn Machine Learning

APRIL 10, 2025

Seamless AWS Integration Works effortlessly with AWS S3 (data storage), AWS Lambda (serverless computing), and AWS Glue (ETL). And also here the best book to start your Machine Learning journey in. SageMaker Pipelines provides automated workflow capabilities for MLOps pipelines. Your team has limited ML expertise.

AWS 59

AWS ML ML Machine Learning 59

Flipboard

JUNE 16, 2025

Next Steps: Transition into data engineering (PySpark, ETL) or machine learning (TensorFlow, PyTorch). Data Pipelines and Orchestration : Familiarity with tools like Airflow (workflow orchestration), Kafka (real-time data processing), and ETL pipelines is critical for creating efficient data workflows. 📣 Want to advertise in AIM?

Python 50

Python AWS Machine Learning Machine Learning 50