KDnuggets

JUNE 19, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Go vs. Python for Modern Data Workflows: Need Help Deciding?

Python 283

Python Natural Language Processing Data Science Machine Learning 283

Flipboard

NOVEMBER 27, 2024

While customers can perform some basic analysis within their operational or transactional databases, many still need to build custom data pipelines that use batch or streaming jobs to extract, transform, and load (ETL) data into their data warehouse for more comprehensive analysis.

ETL 135

ETL Data Warehouse Analytics Analytics 135

KDnuggets

JUNE 26, 2025

Scheduled Analysis Replace the Manual Trigger with a Schedule Trigger to automatically analyze datasets at regular intervals, perfect for monitoring data sources that update frequently. This proactive approach helps you identify data pipeline issues before they impact downstream analysis or model performance.

Data Quality 266

Data Quality Data Science Natural Language Processing Machine Learning 266

IBM Journey to AI blog

MAY 15, 2024

Data must be combined and harmonized from multiple sources into a unified, coherent format before being used with AI models. Unified, governed data can also be put to use for various analytical, operational and decision-making purposes. This process is known as data integration, one of the key components to a strong data fabric.

Data Pipeline 85

Data Pipeline ETL SQL Database 85

IBM Data Science in Practice

JANUARY 13, 2025

The blog post explains how the Internal Cloud Analytics team leveraged cloud resources like Code-Engine to improve, refine, and scale the data pipelines. Background One of the Analytics teams tasks is to load data from multiple sources and unify it into a data warehouse.

ETL 100

ETL Data Pipeline Database Data Warehouse 100

Pickl AI

JULY 8, 2024

Summary: This blog explains how to build efficient data pipelines, detailing each step from data collection to final delivery. Introduction Data pipelines play a pivotal role in modern data architecture by seamlessly transporting and transforming raw data into valuable insights.

Data Pipeline 52

Data Pipeline Data Quality Database Apache Kafka 52

Flipboard

JUNE 18, 2025

Orchestration platform Orq noted in a blog post that AI management systems include four key components: prompt management for consistent model interaction, integration tools, state management and monitoring tools to track performance. What do they need the AI application or agents to do, and how are these planned to support their work?

AI 165

AI AI Data Pipeline ML 165

IBM Journey to AI blog

SEPTEMBER 19, 2023

Though you may encounter the terms “data science” and “data analytics” being used interchangeably in conversations or online, they refer to two distinctly different concepts. Meanwhile, data analytics is the act of examining datasets to extract value and find answers to specific questions.

Data Science 94

Data Science Analytics Analytics Data Scientist 94

phData

AUGUST 6, 2024

As today’s world keeps progressing towards data-driven decisions, organizations must have quality data created from efficient and effective data pipelines. For customers in Snowflake, Snowpark is a powerful tool for building these effective and scalable data pipelines.

Data Pipeline 52

Data Pipeline Python Data Engineering Data Engineering 52

phData

NOVEMBER 4, 2024

The modern data stack is defined by its ability to handle large datasets, support complex analytical workflows, and scale effortlessly as data and business needs grow. Two key technologies that have become foundational for this type of architecture are the Snowflake AI Data Cloud and Dataiku.

Machine Learning 69

Machine Learning Machine Learning Data Science ML 69

Data Science Blog

FEBRUARY 4, 2023

The machine sensor data can be monitored directly in real time via respective data pipelines (real-time stream analytics) or brought into an overall picture of aggregated key figures (reporting). Or maybe you are interested in an individual data strategy ? material flow analysis) for manufacturing and supply chain.

Cloud Data 147

Cloud Data Data Science Business Intelligence Data Pipeline 147

The MLOps Blog

MAY 17, 2023

We also discuss different types of ETL pipelines for ML use cases and provide real-world examples of their use to help data engineers choose the right one. What is an ETL data pipeline in ML? Xoriant It is common to use ETL data pipeline and data pipeline interchangeably.

ETL 59

ETL Data Pipeline ML ML 59

AWS Machine Learning Blog

OCTOBER 24, 2024

Amazon QuickSight powers data-driven organizations with unified (BI) at hyperscale. With QuickSight, all users can meet varying analytic needs from the same source of truth through modern interactive dashboards, paginated reports, embedded analytics, and natural language queries.

Data Warehouse 115

Data Warehouse Machine Learning Machine Learning Cloud Data 115

Pickl AI

DECEMBER 26, 2024

Summary: “Data Science in a Cloud World” highlights how cloud computing transforms Data Science by providing scalable, cost-effective solutions for big data, Machine Learning, and real-time analytics. Advancements in data processing, storage, and analysis technologies power this transformation.

Data Science 52

Data Science Cloud Computing Machine Learning Machine Learning 52

AWS Machine Learning Blog

DECEMBER 4, 2024

SageMaker Unified Studio combines various AWS services, including Amazon Bedrock , Amazon SageMaker , Amazon Redshift , Amazon Glue , Amazon Athena , and Amazon Managed Workflows for Apache Airflow (MWAA) , into a comprehensive data and AI development platform. You’ll use this file when setting up your function to query sales data.

AWS 107

AWS AI AI SQL 107

IBM Journey to AI blog

FEBRUARY 2, 2024

Almost a year ago, IBM encountered a data validation issue during one of our time-sensitive mergers and acquisitions data flows. These changes impact workflows, which in turn affect downstream data pipeline processing, leading to a ripple effect.

Data Pipeline 109

Data Pipeline Data Observability Machine Learning Machine Learning 109

Dataversity

MARCH 13, 2023

Companies are spending a lot of money on data and analytics capabilities, creating more and more data products for people inside and outside the company. These products rely on a tangle of data pipelines, each a choreography of software executions transporting data from one place to another.

Data Observability 98

Data Observability Data Pipeline Analytics Analytics 98

AWS Machine Learning Blog

JANUARY 15, 2025

Understanding customer satisfaction and areas needing improvement from raw data is complex and often requires advanced analytical tools. He has successfully led numerous client engagements to deliver data analytics and AI/machine learning solutions.

AWS 106

AWS SQL AI AI 106

AWS Machine Learning Blog

APRIL 3, 2025

At the heart of this transformation is the OMRON Data & Analytics Platform (ODAP), an innovative initiative designed to revolutionize how the company harnesses its data assets. The robust security features provided by Amazon S3, including encryption and durability, were used to provide data protection.

AWS 90

AWS Data Governance Data Silos SQL 90

Smart Data Collective

SEPTEMBER 8, 2021

The data is initially extracted from a vast array of sources before transforming and converting it to a specific format based on business requirements. ETL is one of the most integral processes required by Business Intelligence and Analytics use cases since it relies on the data stored in Data Warehouses to build reports and visualizations.

ETL 126

ETL Hadoop Data Warehouse Data Pipeline 126

Towards AI

MARCH 21, 2023

If you ever wonder how predictions and forecasts are made based on the raw data collected, stored, and processed in different formats by website feedback, customer surveys, and media analytics, this blog is for you. To learn more about visualizations, you can refer to one of our many blogs on data visualization for a glance.

Data Engineering 98

Data Engineering Data Engineering Data Engineering Data Engineer 98

Smart Data Collective

SEPTEMBER 23, 2020

However, a data lake functions for one specific company, the data warehouse, on the other hand, is fitted for another. This blog will reveal or show the difference between the data warehouse and the data lake. Data Warehouse. Engineers make use of data lakes in storing incoming data.

Data Lakes 135

Data Lakes Data Warehouse Big Data Big Data 135

IBM Journey to AI blog

OCTOBER 22, 2024

Leaders feel the pressure to infuse their processes with artificial intelligence (AI) and are looking for ways to harness the insights in their data platforms to fuel this movement. Indeed, IDC has predicted that by the end of 2024, 65% of CIOs will face pressure to adopt digital tech , such as generative AI and deep analytics.

Data Silos 102

Data Silos Data Pipeline DataOps Business Intelligence 102

Pickl AI

MARCH 19, 2025

Summary: Data engineering tools streamline data collection, storage, and processing. Learning these tools is crucial for building scalable data pipelines. offers Data Science courses covering these tools with a job guarantee for career growth. Below are 20 essential tools every data engineer should know.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

Dataversity

APRIL 1, 2021

In the data-driven world we live in today, the field of analytics has become increasingly important to remain competitive in business. In fact, a study by McKinsey Global Institute shows that data-driven organizations are 23 times more likely to outperform competitors in customer acquisition and nine times […].

Analytics 69

Analytics Analytics Data Silos Data Lakes 69

AWS Machine Learning Blog

FEBRUARY 5, 2025

In this two-part blog post series, we explore the key opportunities OfferUp embraced on their journey to boost and transform their existing search solution from traditional lexical search to modern multimodal search powered by Amazon Bedrock and Amazon OpenSearch Service.

K-nearest Neighbors 89

K-nearest Neighbors Machine Learning Machine Learning Database 89

phData

APRIL 7, 2025

KNIME , a popular open-source data analytics, reporting, and integration platform, offers an excellent solution for implementing low-barrier yet high-value automations that many businesses will find useful with its Business Hub. This platform allows users to create, share, and manage data workflows effortlessly across teams.

Data Pipeline 52

Data Pipeline Analytics Data Analysis Analytics 52

AWS Machine Learning Blog

JANUARY 7, 2025

This blog post with accompanying code presents a solution to experiment with real-time machine translation using foundation models (FMs) available in Amazon Bedrock. It can help collect more data on the value of LLMs for your content translation use cases. He helps customers in the Northeast U.S.

AWS 115

AWS Python AI AI 115

phData

MAY 25, 2023

How to Optimize Power BI and Snowflake for Advanced Analytics Spencer Baucke May 25, 2023 The world of business intelligence and data modernization has never been more competitive than it is today. Much of what is discussed in this guide will assume some level of analytics strategy has been considered and/or defined. No problem!

Power BI 84

Power BI Analytics Analytics Azure 84

AWS Machine Learning Blog

JULY 3, 2025

This following diagram illustrates the enhanced data extract, transform, and load (ETL) pipeline interaction with Amazon Bedrock. To achieve the desired accuracy in KPI calculations, the data pipeline was refined to achieve consistent and precise performance, which leads to meaningful insights.

AWS 70

AWS AI AI SQL 70

phData

MAY 3, 2024

Leveraging real-time analytics to make informed decisions is the golden standard for virtually every business that collects data. If you have the Snowflake Data Cloud (or are considering migrating to Snowflake ), you’re a blog away from taking a step closer to real-time analytics.

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

Data Science Dojo

APRIL 16, 2024

In this blog, we will explore the top 10 AI jobs and careers that are also the highest-paying opportunities for individuals in 2024. Big data engineer Potential pay range – US$206,000 to 296,000/yr They operate at the backend to build and maintain complex systems that store and process the vast amounts of data that fuel AI applications.

AI 195

AI AI Machine Learning Machine Learning 195

phData

FEBRUARY 24, 2023

Alteryx and the Snowflake Data Cloud offer a potential solution to this issue and can speed up your path to Analytics. In this blog post, we will explore how Alteryx and Snowflake can accelerate your journey to Analytics by sharing use cases and best practices. What is Alteryx? What is Snowflake?

Analytics 52

Analytics Analytics Database Python 52

AWS Machine Learning Blog

FEBRUARY 21, 2025

Previously, he was a Data & Machine Learning Engineer at AWS, where he worked closely with customers to develop enterprise-scale data infrastructure, including data lakes, analytics dashboards, and ETL pipelines. He specializes in designing, building, and optimizing large-scale data solutions.

AWS 127

AWS Natural Language Processing Machine Learning Machine Learning 127

phData

SEPTEMBER 5, 2023

Google Analytics 4 (GA4) is a powerful tool for collecting and analyzing website and app data that many businesses rely heavily on to make informed business decisions. However, there might be instances where you need to migrate the raw event data from GA4 to Snowflake for more in-depth analysis and business intelligence purposes.

Azure 52

Azure Analytics Analytics Data Pipeline 52

AWS Machine Learning Blog

FEBRUARY 24, 2023

Solution overview In brief, the solution involved building three pipelines: Data pipeline – Extracts the metadata of the images Machine learning pipeline – Classifies and labels images Human-in-the-loop review pipeline – Uses a human team to review results The following diagram illustrates the solution architecture.

ML 101

ML ML AWS Data Pipeline 101

IBM Journey to AI blog

FEBRUARY 26, 2024

Increased data pipeline observability As discussed above, there are countless threats to your organization’s bottom line. That’s why data pipeline observability is so important. Realize the benefits of automated data lineage today. Schedule a demo with a MANTA engineer to learn more.

Data Pipeline 112

Data Pipeline Data Governance Data Engineering Data Engineering 112