KDnuggets

JUNE 6, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter 5 Error Handling Patterns in Python (Beyond Try-Except) Stop letting errors crash your app.

Python 166

Python Natural Language Processing Data Science Machine Learning 166

KDnuggets

JUNE 9, 2025

Blog Top Posts About Topics AI Career Advice Computer Vision Data Engineering Data Science Language Models Machine Learning MLOps NLP Programming Python SQL Datasets Events Resources Cheat Sheets Recommendations Tech Briefs Advertise Join Newsletter Run the Full DeepSeek-R1-0528 Model Locally Running the quantized version DeepSeek-R1-0528 Model locally (..)

Natural Language Processing 143

Natural Language Processing Data Science Machine Learning Machine Learning 143

AWS Machine Learning Blog

FEBRUARY 18, 2025

During these live events, F1 IT engineers must triage critical issues across its services, such as network degradation to one of its APIs. An Amazon EventBridge schedule checked this bucket hourly for new files and triggered log transformation extract, transform, and load (ETL) pipelines built using AWS Glue and Apache Spark.

AWS 77

AWS Database ETL AI 77

Data Science Dojo

JULY 3, 2024

Databases and SQL : Managing and querying relational databases using SQL, as well as working with NoSQL databases like MongoDB. Data Engineering : Building and maintaining data pipelines, ETL (Extract, Transform, Load) processes, and data warehousing.

Data Science 150

Data Science Machine Learning Machine Learning Data Visualization 150

Smart Data Collective

APRIL 29, 2020

Extraction, Transform, Load (ETL). Redshift is the product for data warehousing, and Athena provides SQL data analytics. Profisee notices changes in data and assigns events within the systems. Dataform is a data transformation platform that is based on SQL. Master data management. Data transformation.

Data Warehouse 98

Data Warehouse Azure SQL ETL 98

AWS Machine Learning Blog

FEBRUARY 21, 2025

Apache Hive was used to provide a tabular interface to data stored in HDFS, and to integrate with Apache Spark SQL. The diagram depicts the flow; the key components are detailed below: Data Ingestion: Data is ingested into the system using Attunity data ingestion in Spark SQL. Analytic data is stored in Amazon Redshift.

Data Science 85

Data Science AWS Hadoop Data Scientist 85

Smart Data Collective

JULY 20, 2021

This tool is designed to connect various data sources, enterprise applications and perform analytics and ETL processes. This ETL integration software allows you to build integrations anytime and anywhere without requiring any coding. Moreover, it allows you to explore the data in SQL and view it in any analytics tool efficiently.

Big Data 142

Big Data Big Data ETL Analytics 142

AWS Machine Learning Blog

JANUARY 17, 2024

It can represent a geographical area as a whole or it can represent an event associated with a geographical area. To obtain such insights, the incoming raw data goes through an extract, transform, and load (ETL) process to identify activities or engagements from the continuous stream of device location pings.

Clustering 137

Clustering AWS ML ML 137

Pickl AI

MARCH 19, 2025

Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Python, SQL, and Apache Spark are essential for data engineering workflows. SQL Structured Query Language ( SQL ) is a fundamental skill for data engineers.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

IBM Journey to AI blog

AUGUST 29, 2024

Event-driven businesses across all industries thrive on real-time data, enabling companies to act on events as they happen rather than after the fact. This is where Apache Flink shines, offering a powerful solution to harness the full potential of an event-driven business model through efficient computing and processing capabilities.

Apache Kafka 45

Apache Kafka Hadoop ETL Data Pipeline 45

phData

APRIL 29, 2024

Understanding Fivetran Fivetran is a popular Software-as-a-Service platform that enables users to automate the movement of data and ETL processes across diverse sources to a target destination. For a longer overview, along with insights and best practices, please feel free to jump back to the previous blog.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

ODSC - Open Data Science

SEPTEMBER 27, 2023

Data Warehouses Some key characteristics of data warehouses are as follows: Data Type: Data warehouses primarily store structured data that has undergone ETL (Extract, Transform, Load) processing to conform to a specific schema. Processing: Relational databases are optimized for transactional processing and structured queries using SQL.

Data Lakes 98

Data Lakes Data Warehouse Database Big Data 98

Smart Data Collective

MAY 20, 2019

The entire process is also achieved much faster, boosting not just general efficiency but an organization’s reaction time to certain events, as well. The popular tools, on the other hand, include Power BI, ETL, IBM Db2, and Teradata. For frameworks and languages, there’s SAS, Python, R, Apache Hadoop and many others.

Analytics 111

Analytics Analytics Data Analyst Machine Learning 111

AWS Machine Learning Blog

JANUARY 6, 2023

The rules in this engine were predefined and written in SQL, which aside from posing a challenge to manage, also struggled to cope with the proliferation of data from TR’s various integrated data source. As the users are interacting with TR’s applications, they generate clickstream events, which are published into Amazon Kinesis Data Streams.

AWS 98

AWS Data Warehouse ML ML 98

ODSC - Open Data Science

JANUARY 18, 2024

EVENT — ODSC East 2024 In-Person and Virtual Conference April 23rd to 25th, 2024 Join us for a deep dive into the latest data science and AI trends, tools, and techniques, from LLMs to data analytics and from machine learning to responsible AI. Interested in attending an ODSC event? Learn more about our upcoming events here.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

OCTOBER 20, 2023

Some of the databases supported by Fivetran are: Snowflake Data Cloud (BETA) MySQL PostgreSQL SAP ERP SQL Server Oracle In this blog, we will review how to pull Data from on-premise Systems using Fivetran to a specific target or destination. The most common example of such databases is where events are tracked.

Database 52

Database SQL ETL Data Warehouse 52

The MLOps Blog

MARCH 15, 2023

Data Ingestion : Involves raw data collection from origin and storage using architectures such as batch, streaming or event-driven. Enables users to trigger their custom transformations via SQL and dbt. Relational database connectors such as Teradata, Oracle, and Microsoft SQL servers are available.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

Pickl AI

SEPTEMBER 19, 2024

It also supports ETL (Extract, Transform, Load) processes, making data warehousing and analytics essential. Spark SQL Spark SQL is a module that works with structured and semi-structured data. It allows users to run SQL queries, read data from different sources, and seamlessly integrate with Spark’s core capabilities.

Hadoop 52

Hadoop Big Data Big Data Clustering 52

ODSC - Open Data Science

SEPTEMBER 29, 2023

Spark is more focused on data science, ingestion, and ETL, while HPCC Systems focuses on ETL and data delivery and governance. It’s not a widely known programming language like Java, Python, or SQL. ECL sounds compelling, but it is a new programming language and has fewer users than languages like Python or SQL.

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

Dataconomy

MARCH 24, 2023

BI developer:  A BI developer is responsible for designing and implementing BI solutions, including data warehouses, ETL processes, and reports. Database management:  A BI professional should be able to design and manage databases, including data modeling, ETL processes, and data integration.

Business Intelligence 70

Business Intelligence Business Intelligence Data Analysis Data Analysis 70

Dataconomy

MARCH 24, 2023

BI developer:  A BI developer is responsible for designing and implementing BI solutions, including data warehouses, ETL processes, and reports. Database management:  A BI professional should be able to design and manage databases, including data modeling, ETL processes, and data integration.

Business Intelligence 70

Business Intelligence Business Intelligence Data Analysis Data Analysis 70

phData

OCTOBER 17, 2024

Apache Airflow Airflow is an open-source ETL software that is very useful when paired with Snowflake. The DAGs can then be scheduled to run at specific intervals or triggered when an event occurs. Airflow uses Directed Acyclic Graphs (DAGs) to represent workflows as tasks with defined dependencies.

AI 52

AI AI SQL Data Quality 52

phData

NOVEMBER 30, 2023

For example, you can use alerts to send notifications, capture data, or execute SQL commands when certain events or thresholds are reached in your data. A task is a SQL statement that runs on a schedule or when triggered by other tasks. SQL commands allow users to create, modify, suspend, resume, and drop tasks.

SQL 52

SQL Cloud Data Database ETL 52

Pickl AI

NOVEMBER 4, 2024

Key components of data warehousing include: ETL Processes: ETL stands for Extract, Transform, Load. ETL is vital for ensuring data quality and integrity. Apache Kafka Kafka is a distributed event streaming platform for building real-time data pipelines and streaming applications.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

Tableau

AUGUST 21, 2023

Hyper Supercharge your analytics with in-memory data engine Hyper is Tableau's blazingly fast SQL engine that lets you do fast real-time analytics, interactive exploration, and ETL transformations through Tableau Prep. An ODBC connector lets you access any data source that supports the SQL standard and implements the ODBC API.

Tableau 97

Tableau Database Analytics Analytics 97

phData

FEBRUARY 5, 2024

The tool converts the templated configuration into a set of SQL commands that are executed against the target Snowflake environment. Instead of manually converting these queries, consider using software built to automate the translation of queries from your legacy systems language to Snowflake’s version, such as phData’s SQL Translation Tool.

SQL 52

SQL Database Data Quality Data Warehouse 52

Alation

OCTOBER 27, 2022

In celebration of last week’s dbt Coalesce, their flagship event, I interviewed the D&A team to learn more about how they leverage dbt to support excellence in analytics. Adrian : Fivetran and dbt enable us to easily connect data sources and write SQL transformations to power downstream dashboards and reporting.

Data Analyst 52

Data Analyst Data Scientist Analytics Analytics 52

ODSC - Open Data Science

DECEMBER 9, 2024

Switching contexts across tools like Pandas, SciKit-Learn, SQL databases, and visualization engines creates cognitive burden. For organizations beginning the journey, an incremental approach allows quick wins while building internal expertise over time through online education, community events, andmentors.

Data Science 40

Data Science Python Machine Learning Machine Learning 40

Pickl AI

JANUARY 22, 2025

This involves selecting appropriate Database Management Systems (DBMS) such as Oracle, SQL Server, or MySQL. the event of data loss, DBAs are responsible for restoring databases from backups efficiently to minimize downtime. Their expertise is crucial in projects involving data extraction, transformation, and loading (ETL) processes.

Database 52

Database Database Administration ETL SQL 52

Pickl AI

AUGUST 11, 2024

These tables are called “factless fact tables” or “junction tables” They are used for modelling many-to-many relationships or for capturing timestamps of events. A star schema forms when a fact table combines with its dimension tables. This schema serves as the foundation of dimensional modeling.

Data Quality 52

Data Quality Data Warehouse Data Governance Analytics 52

phData

FEBRUARY 14, 2023

Fivetran Fivetran is a tool dedicated to replicating applications, databases, events, and files into a high-performance data warehouse, such as Snowflake. To help you make your choice, here are the ones we consider to be the best. What Are the Best Third-Party Data Ingestion Tools for Snowflake?

Data Warehouse 59

Data Warehouse Azure AWS Database 59

Flipboard

MARCH 21, 2025

Traditionally, answering this question would involve multiple data exports, complex extract, transform, and load (ETL) processes, and careful data synchronization across systems. Users can write data to managed RMS tables using Iceberg APIs, Amazon Redshift, or Zero-ETL ingestion from supported data sources.

SQL 140

SQL Data Analyst Data Warehouse AWS 140

Pickl AI

AUGUST 9, 2024

Understanding the differences between SQL and NoSQL databases is crucial for students. Understanding ETL (Extract, Transform, Load) processes is vital for students. Students should understand the concepts of event-driven architecture and stream processing. Knowledge of RESTful APIs and authentication methods is essential.

Big Data 52

Big Data Big Data Big Data Analytics Big Data Analytics 52

DagsHub

APRIL 7, 2024

Thanks to its various operators, it is integrated with Python, Spark, Bash, SQL, and more. Flexibility: Its use cases are wider than just machine learning; for example, we can use it to set up ETL pipelines. Flexibility: Airflow was designed with batch workflows in mind; it was not meant for permanently running event-based workflows.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

DagsHub

OCTOBER 23, 2024

Here’s the structured equivalent of this same data in tabular form: With structured data, you can use query languages like SQL to extract and interpret information. Apache Kafka Apache Kafka is a distributed event streaming platform for real-time data pipelines and stream processing. Unstructured.io

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

phData

OCTOBER 28, 2024

Modern low-code/no-code ETL tools allow data engineers and analysts to build pipelines seamlessly using a drag-and-drop and configure approach with minimal coding. One such option is the availability of Python Components in Matillion ETL, which allows us to run Python code inside the Matillion instance.

Python 52

Python ETL AWS Database 52

phData

SEPTEMBER 27, 2024

In this guide, we will explore concepts like transitional modeling for customer profiles, the power of event logs for customer behavior, persistent staging for raw customer data, real-time customer data capture, and much more. Rich Context: Each event carries with it a wealth of contextual information. What is Activity Schema Modeling?

Data Modeling 52

Data Modeling Data Models Apache Kafka Data Lakes 52