AWS Machine Learning Blog

FEBRUARY 7, 2025

It is backed by Amazon Managed Streaming for Apache Kafka (Amazon MSK) (8). The resources in the Kubernetes cluster are deployed in a private subnet. The central place for Knative Eventing is the Knative broker (7).

Analytics 66

Analytics Analytics AWS Clustering 66

Hacker News

JULY 1, 2025

My ideal role/level is as a team lead who gets to ship code.

Python 57

Python AWS SQL ML 57

Pickl AI

MARCH 13, 2025

Two of the most popular message brokers are RabbitMQ and Apache Kafka. In this blog, we will explore RabbitMQ vs Kafka, their key differences, and when to use each. Understanding Apache Kafka Apache Kafka is an open-source system designed to handle real-time data streaming.

Apache Kafka 52

Apache Kafka Big Data Big Data Data Pipeline 52

Pickl AI

MARCH 19, 2025

Python, SQL, and Apache Spark are essential for data engineering workflows. Real-time data processing with Apache Kafka enables faster decision-making. Apache Spark Apache Spark is a powerful data processing framework that efficiently handles Big Data. The global Big Data and data engineering market, valued at $75.55

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

Pickl AI

FEBRUARY 23, 2025

Best Big Data Tools Popular tools such as Apache Hadoop, Apache Spark, Apache Kafka, and Apache Storm enable businesses to store, process, and analyse data efficiently. Apache Kafka Overview Apache Kafka is an open-source stream-processing platform capable of handling trillions of events per day.

Big Data 52

Big Data Big Data Apache Hadoop Apache Kafka 52

Flipboard

JUNE 3, 2025

The data is then transmitted to Amazon Managed Streaming for Apache Kafka (Amazon MSK) to facilitate high-throughput, reliable streaming. Data ingestion and processing EV charging stations send real-time charging data to AWS IoT Core , which acts as the initial entry point for data processing.

AWS 139

AWS Apache Kafka Database AI 139

Analytics Vidhya

JULY 22, 2022

That’s why you need to know about Apache Kafka, a publish-subscribe messaging system you can use to build distributed applications. The post Apache Kafka Architecture and Use Cases Explained appeared first on Analytics Vidhya. It is scalable and fault-tolerant, making […].

Apache Kafka 336

Apache Kafka Big Data Big Data Data Science 336

Analytics Vidhya

JUNE 21, 2022

The post Handling Streaming Data with Apache Kafka – A First Look appeared first on Analytics Vidhya. Streaming Data is generated continuously, by multiple data sources say, sensors, server logs, stock prices, etc. These records are usually small and in the order […].

Apache Kafka 379

Apache Kafka Data Science Analytics Analytics 379

Analytics Vidhya

OCTOBER 3, 2022

The post Apache Kafka Use Cases and Installation Guide appeared first on Analytics Vidhya. As applications cover more aspects of our daily lives, it is increasingly difficult to provide users with a quick response. Source: kafka.apache.org Caching is used to solve […].

Apache Kafka 343

Apache Kafka Data Science Analytics Analytics 343

Analytics Vidhya

AUGUST 2, 2022

Introduction Earlier, I had introduced basic concepts of Apache Kafka in my blog on Analytics Vidhya(link is available under references). This article introduced concepts involved in Apache Kafka and further built the understanding by using the python API of Kafka to write some […].

Apache Kafka 290

Apache Kafka Data Science Python Analytics 290

Analytics Vidhya

DECEMBER 30, 2022

The post Introduction to Apache Kafka: Fundamentals and Working appeared first on Analytics Vidhya. Introduction Have you ever wondered how Instagram recommends similar kinds of reels while you are scrolling through your feed or ad recommendations for similar products that you were browsing on Amazon?

Apache Kafka 315

Apache Kafka Data Science Analytics Analytics 315

Analytics Vidhya

MARCH 10, 2023

Introduction Apache Kafka is a framework for dealing with many real-time data streams in a way that is spread out. It was made on LinkedIn and shared with the public in 2011.

Apache Kafka 306

Apache Kafka Data Pipeline Analytics Analytics 306

Analytics Vidhya

APRIL 28, 2023

Introduction Apache Kafka is an open-source publish-subscribe messaging application initially developed by LinkedIn in early 2011. It is a famous Scala-coded data processing tool that offers low latency, extensive throughput, and a unified platform to handle the data in real-time.

Apache Kafka 289

Apache Kafka Analytics Analytics Hadoop 289

Analytics Vidhya

NOVEMBER 2, 2020

Overview Learn about viewing data as streams of immutable events in contrast to mutable containers Understand how Apache Kafka captures real-time data through event. The post Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers appeared first on Analytics Vidhya.

Apache Kafka 291

Apache Kafka Data Scientist Data Engineering Data Engineering 291

KDnuggets

APRIL 5, 2023

Learn about Apache Kafka architecture and its implementation using a real-world use case of a taxi booking app.

Apache Kafka 270

Apache Kafka Data Engineering Data Engineering Data Engineering 270

KDnuggets

APRIL 1, 2025

This article explains how to create a system that processes data in real time using Apache Kafka and Spark.

Apache Kafka 212

Apache Kafka Data Science Analytics Analytics 212

Dataconomy

MAY 26, 2017

The post Amazon Kinesis vs. Apache Kafka For Big Data Analysis appeared first on Dataconomy. Data processing today is done in form of pipelines which include various steps like aggregation, sanitization, filtering and finally generating insights by applying various statistical models. Parts of the Kinesis platform are.

Apache Kafka 155

Apache Kafka Big Data Big Data Data Analysis 155

databricks

AUGUST 12, 2024

The blog explores data streams from NASA satellites using Apache Kafka and Databricks. It demonstrates ingestion and transformation with Delta Live Tables in SQL and AI/BI-powered analysis of supernova events.

Apache Kafka 284

Apache Kafka SQL AI AI 284

Analytics Vidhya

SEPTEMBER 22, 2022

Dale Carnegie” Apache Kafka is a Software Framework for storing, reading, and analyzing streaming data. This article was published as a part of the Data Science Blogathon. Introduction “Learning is an active process. We learn by doing. Only knowledge that is used sticks in your mind.-

Data Pipeline 270

Data Pipeline Apache Kafka Internet of Things Data Science 270

Hacker News

JUNE 12, 2025

Available Service information One or more regions affected Products Americas (regions) Europe (regions) Asia Pacific (regions) Middle East (regions) Africa (regions) Multi-regions Global Access Approval Access Context Manager Access Transparency Agent Assist AI Platform Prediction AI Platform Training AlloyDB for PostgreSQL Anthos Service Mesh API (..)

Apache Kafka 140

Apache Kafka Cloud Data AI AI 140

Analytics Vidhya

JULY 12, 2023

Best Big Data Softwares - Apache Hadoop, Apache Spark, apache Kafka, Apache Storm, Apache Cassandra, Apache Hive, zoho & more.

Apache Kafka 265

Apache Kafka Apache Hadoop Big Data Big Data 265

Hacker News

FEBRUARY 8, 2023

Learn what windowing is, the difference between the four types of windows (hopping and tumbling, or session and sliding), and how to create them.

Apache Kafka 174

Apache Kafka 174

Hacker News

FEBRUARY 5, 2024

A desktop client for Apache Kafka. Contribute to Bogdanp/Franz development by creating an account on GitHub.

Apache Kafka 105

Apache Kafka 105

KDnuggets

APRIL 12, 2023

How to Build a Scalable Data Architecture with Apache Kafka Top 19 Skills You Need to Know in 2023 to Be a Data Scientist • 8 Open-Source Alternative to ChatGPT and Bard • Free eBook: 10 Practical Python Programming Tricks • DataLang: A New Programming Language for Data Scientists… Created by ChatGPT? •

Apache Kafka 223

Apache Kafka Data Scientist Python 223

IBM Journey to AI blog

FEBRUARY 12, 2024

At the forefront of this event-driven revolution is Apache Kafka, the widely recognized and dominant open-source technology for event streaming. While most enterprises have already recognized how Apache Kafka provides a strong foundation for EDA, they often fall behind in unlocking its true potential.

Apache Kafka 100

Apache Kafka EDA SQL Database 100

Hacker News

JULY 13, 2023

While playing Factorio the other day, I was struck by the many similarities with Apache Kafka.

Apache Kafka 179

Apache Kafka 179

IBM Journey to AI blog

NOVEMBER 3, 2023

Apache Kafka and Apache Flink working together Anyone who is familiar with the stream processing ecosystem is familiar with Apache Kafka: the de-facto enterprise standard for open-source event streaming. With Apache Kafka, you get a raw stream of events from everything that is happening within your business.

Apache Kafka 79

Apache Kafka Data Warehouse Data Pipeline Big Data 79

Hacker News

OCTOBER 22, 2023

The choice between OpenTelemetry Collector and Apache Kafka isn't a zero-sum game. Each has its unique strengths and can even complement each other in certain architectures.

Apache Kafka 176

Apache Kafka Data Quality 176

Hacker News

AUGUST 30, 2024

This is a guest article by Stanislav Kozlovski, an Apache Kafka Committer. If you would like to connect with Stanislav, you can do so on Twitter and LinkedIn. AWS S3 is a service every engineer is familiar with. It’s the service that popularized the notion of cold-storage to the

Apache Kafka 152

Apache Kafka AWS 152

IBM Journey to AI blog

SEPTEMBER 4, 2024

Apache Kafka is an open-source , distributed streaming platform that allows developers to build real-time, event-driven applications. With Apache Kafka, developers can build applications that continuously use streaming data records and deliver real-time experiences to users. How does Apache Kafka work?

Apache Kafka 58

Apache Kafka Internet of Things Data Pipeline Clustering 58

Hacker News

APRIL 7, 2024

A cloud native implementation for Apache Kafka, reducing your cloud infrastructure bill by up to 90%. AutoMQ/automq

Apache Kafka 134

Apache Kafka 134

Smart Data Collective

AUGUST 17, 2022

You can safely use an Apache Kafka cluster for seamless data movement from the on-premise hardware solution to the data lake using various cloud services like Amazon’s S3 and others. 5 Key Comparisons in Different Apache Kafka Architectures. 5 Key Comparisons in Different Apache Kafka Architectures.

Apache Kafka 128

Apache Kafka ETL Data Lakes AWS 128

Hacker News

DECEMBER 18, 2023

Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs. Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs. A multi-protocol, event-native proxy. GitHub - aklivity/zilla: ?

Apache Kafka 98

Apache Kafka 98

Hacker News

JUNE 15, 2023

Comments (..)

Apache Kafka 99

Apache Kafka 99

ODSC - Open Data Science

MAY 31, 2023

Be sure to check out his talk, “ Apache Kafka for Real-Time Machine Learning Without a Data Lake ,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem.

Data Lakes 131

Data Lakes Machine Learning Machine Learning Apache Kafka 131

Hacker News

MAY 8, 2024

Recently I wanted to learn a bit about Apache Kafka. It is often used as a way to do event sourcing (or similar message-driven architectures). An “add-on” to.

Apache Kafka 105

Apache Kafka 105

Towards AI

FEBRUARY 29, 2024

Within this article, we will explore the significance of these pipelines and utilise robust tools such as Apache Kafka and Spark to manage vast streams of data efficiently. Apache Kafka Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications.

Apache Kafka 142

Apache Kafka SQL Clustering Data Pipeline 142