How to Build a Scalable Data Architecture with Apache Kafka
KDnuggets
APRIL 5, 2023
Learn about Apache Kafka architecture and its implementation using a real-world use case of a taxi booking app.
Apache Kafka Related Topics 
Apache Kafka and Apache Flink: An open-source match made in heaven
IBM Journey to AI blog
NOVEMBER 3, 2023
Apache Kafka and Apache Flink working together Anyone who is familiar with the stream processing ecosystem is familiar with Apache Kafka: the de-facto enterprise standard for open-source event streaming. With Apache Kafka, you get a raw stream of events from everything that is happening within your business.
Apache Kafka 
75
75 
Build a Scalable Data Pipeline with Apache Kafka
Analytics Vidhya
MARCH 10, 2023
Introduction Apache Kafka is a framework for dealing with many real-time data streams in a way that is spread out. It was made on LinkedIn and shared with the public in 2011.
Apache Kafka 95
95 
A Detailed Guide of Interview Questions on Apache Kafka
Analytics Vidhya
APRIL 28, 2023
Introduction Apache Kafka is an open-source publish-subscribe messaging application initially developed by LinkedIn in early 2011. It is a famous Scala-coded data processing tool that offers low latency, extensive throughput, and a unified platform to handle the data in real-time.
Apache Kafka 83
83 
Apache Kafka Architecture and Use Cases Explained
Analytics Vidhya
JULY 22, 2022
That’s why you need to know about Apache Kafka, a publish-subscribe messaging system you can use to build distributed applications. The post Apache Kafka Architecture and Use Cases Explained appeared first on Analytics Vidhya. It is scalable and fault-tolerant, making […].
Apache Kafka 102
102 
Apache Kafka Use Cases and Installation Guide
Analytics Vidhya
OCTOBER 3, 2022
The post Apache Kafka Use Cases and Installation Guide appeared first on Analytics Vidhya. As applications cover more aspects of our daily lives, it is increasingly difficult to provide users with a quick response. Source: kafka.apache.org Caching is used to solve […].
Apache Kafka 104
104 
Exploring Partitions and Consumer Groups in Apache Kafka
Analytics Vidhya
AUGUST 2, 2022
Introduction Earlier, I had introduced basic concepts of Apache Kafka in my blog on Analytics Vidhya(link is available under references). This article introduced concepts involved in Apache Kafka and further built the understanding by using the python API of Kafka to write some […].
Apache Kafka 87
87 
Introduction to Apache Kafka: Fundamentals and Working
Analytics Vidhya
DECEMBER 30, 2022
The post Introduction to Apache Kafka: Fundamentals and Working appeared first on Analytics Vidhya. Introduction Have you ever wondered how Instagram recommends similar kinds of reels while you are scrolling through your feed or ad recommendations for similar products that you were browsing on Amazon?
Apache Kafka 88
88 
Handling Streaming Data with Apache Kafka – A First Look
Analytics Vidhya
JUNE 21, 2022
The post Handling Streaming Data with Apache Kafka – A First Look appeared first on Analytics Vidhya. Streaming Data is generated continuously, by multiple data sources say, sensors, server logs, stock prices, etc. These records are usually small and in the order […].
Apache Kafka 118
118 
Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers
Analytics Vidhya
NOVEMBER 2, 2020
Overview Learn about viewing data as streams of immutable events in contrast to mutable containers Understand how Apache Kafka captures real-time data through event. The post Apache Kafka: A Metaphorical Introduction to Event Streaming for Data Scientists and Data Engineers appeared first on Analytics Vidhya.
Apache Kafka 103
103 
Apache Kafka Beyond the Basics: Windowing
Hacker News
FEBRUARY 8, 2023
Learn what windowing is, the difference between the four types of windows (hopping and tumbling, or session and sliding), and how to create them.
Apache Kafka 134
134 Apache Kafka 134
134 
OpenTelemetry at Scale: Using Kafka to handle bursty traffic
Hacker News
OCTOBER 22, 2023
The choice between OpenTelemetry Collector and Apache Kafka isn't a zero-sum game. Each has its unique strengths and can even complement each other in certain architectures.
Apache Kafka 142
142 
KDnuggets News, April 12: Top 19 Skills for a Data Scientist in 2023 • 8 ChatGPT Open-Source Alternatives
KDnuggets
APRIL 12, 2023
How to Build a Scalable Data Architecture with Apache Kafka Top 19 Skills You Need to Know in 2023 to Be a Data Scientist • 8 Open-Source Alternative to ChatGPT and Bard • Free eBook: 10 Practical Python Programming Tricks • DataLang: A New Programming Language for Data Scientists… Created by ChatGPT? •
Data Scientist 170
170 
Understanding Kafka with Factorio (2019)
Hacker News
JULY 13, 2023
While playing Factorio the other day, I was struck by the many similarities with Apache Kafka.
Apache Kafka 141
141 Apache Kafka 141
141 
Streaming Machine Learning Without a Data Lake
ODSC - Open Data Science
MAY 31, 2023
Be sure to check out his talk, “ Apache Kafka for Real-Time Machine Learning Without a Data Lake ,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem.
Data Lakes 131
131 
Hybrid Vs. Multi-Cloud: 5 Key Comparisons in Kafka Architectures
Smart Data Collective
AUGUST 17, 2022
You can safely use an Apache Kafka cluster for seamless data movement from the on-premise hardware solution to the data lake using various cloud services like Amazon’s S3 and others. 5 Key Comparisons in Different Apache Kafka Architectures. 5 Key Comparisons in Different Apache Kafka Architectures.
Apache Kafka 103
103 
Amazon Kinesis vs. Apache Kafka For Big Data Analysis
Dataconomy
MAY 26, 2017
The post Amazon Kinesis vs. Apache Kafka For Big Data Analysis appeared first on Dataconomy. Data processing today is done in form of pipelines which include various steps like aggregation, sanitization, filtering and finally generating insights by applying various statistical models. Parts of the Kinesis platform are.
Apache Kafka 62
62 
Top 15 Big Data Softwares to Know About in 2023
Analytics Vidhya
JULY 12, 2023
Best Big Data Softwares - Apache Hadoop, Apache Spark, apache Kafka, Apache Storm, Apache Cassandra, Apache Hive, zoho & more.
Apache Kafka 73
73 
Level up your Kafka applications with schemas
IBM Journey to AI blog
NOVEMBER 21, 2023
Apache Kafka is a well-known open-source event store and stream processing platform and has grown to become the de facto standard for data streaming. Apache Kafka transfers data without validating the information in the messages.
Apache Kafka 109
109 
Build a Simple Realtime Data Pipeline
Analytics Vidhya
SEPTEMBER 22, 2022
Dale Carnegie” Apache Kafka is a Software Framework for storing, reading, and analyzing streaming data. This article was published as a part of the Data Science Blogathon. Introduction “Learning is an active process. We learn by doing. Only knowledge that is used sticks in your mind.-
Gently Down the Stream – A gentle introduction to Apache Kafka (2021)
Hacker News
JUNE 15, 2023
Comments (..)
Apache Kafka 77
77 Apache Kafka 77
77 
Accelerate your speed of business with IBM Event Automation
IBM Journey to AI blog
MAY 9, 2023
IBM Event Automation provides an intuitive and integrated experience for distributing, discovering and processing business events across the organization: Event distribution: Collect raw streams of real-time business events with enterprise-grade Apache Kafka.
Apache Kafka 106
106 
Event-driven architecture, Kafka and CDPs: Joining internal infrastructure with your tech stack
Twilio Segment
SEPTEMBER 7, 2021
Event streaming platforms such as Apache Kafka are gaining in importance across all industries. In this article we'll discuss the benefits Apache Kafka implementations can gain from pairing it with a CDP.
Apache Kafka 40
40 Apache Kafka 40
40 
Five scalability pitfalls to avoid with your Kafka application
IBM Journey to AI blog
NOVEMBER 9, 2023
Apache Kafka is a high-performance, highly scalable event streaming platform. To unlock Kafka’s full potential, you need to carefully consider the design of your application. It’s all too easy to write Kafka applications that perform poorly or eventually hit a scalability brick wall.
Apache Kafka 70
70 
Memphis: A game changer in the world of traditional messaging systems
Data Science Dojo
MARCH 9, 2023
Challenges for individuals Traditional messaging brokers, such as Apache Kafka, RabbitMQ, and ActiveMQ, have been widely used to enable communication between applications and services. Handling too many data sources can become overwhelming, especially with complex schemas. Debugging and troubleshooting can also be challenging.
Apache Kafka 86
86 
Unlocking Real-Time Mainframe Data Replication with the Precisely Data Integrity Suite and Confluent Data Streams
Precisely
JULY 21, 2023
Used by more than 75% of the Fortune 500, Apache Kafka has emerged as a powerful open source data streaming platform to meet these challenges. But harnessing and integrating Kafka’s full potential into enterprise environments can be complex. This is where Confluent steps in.
Apache Kafka 52
52 
Know Before You Go: Precisely at Confluent’s Current 2023
Precisely
SEPTEMBER 12, 2023
Precisely data integrity solutions fuel your Confluent and Apache Kafka streaming data pipelines with trusted data that has maximum accuracy, consistency, and context and we’re ready to share more with you at the upcoming Current 2023. Let’s cover some additional information to know before attending.
Apache Kafka 52
52 
Big Data – Lambda or Kappa Architecture?
Data Science Blog
JUNE 27, 2023
In practical implementation, the Kappa architecture is commonly deployed using Apache Kafka or Kafka-based tools. Applications can directly read from and write to Kafka or an alternative message queue tool. This architectural concept relies on event streaming as the core element of data delivery.
Big Data 130
130 
All of the Free Virtual Sessions Coming to ODSC Europe 2023
ODSC - Open Data Science
JUNE 7, 2023
Bilokon | Visiting Lecturer, CEO and Founder | Imperial College London, Thalesians Ltd Apache Kafka for Real-Time Machine Learning Without a Data Lake: Kai Waehner | Global Field CTO, Author, International Speaker Semantic Analysis and Procedural Language Understanding in the Era of Large Language Models: Dr.
Apache Kafka 52
52 
Real-time artificial intelligence and event processing
IBM Journey to AI blog
NOVEMBER 29, 2023
IBM Event Automation is a fully composable solution, built on open technologies, with capabilities for: Event streaming : Collect and distribute raw streams of real-time business events with enterprise-grade Apache Kafka. Event endpoint management : Describe and document events easily according to the Async API specification.
Building a Pizza Delivery Service with a Real-Time Analytics Stack
ODSC - Open Data Science
JUNE 1, 2023
We’re going to assume that the pizza service already captures orders in Apache Kafka and is also keeping a record of its customers and the products that they sell in MySQL. Apache Pinot is a real-time OLAP database built at LinkedIn to deliver scalable real-time analytics with low latency.
Analytics 98
98 
Pictures and Highlights from ODSC Europe 2023
ODSC - Open Data Science
JULY 22, 2023
Leverage Compound Sparsity to Achieve the Fastest Inference Performance on CPUs: Damian Bogunowicz | Neural Magic and Konstantin Gulin | Machine Learning Engineer | Neural Magic Apache Kafka for Real-Time Machine Learning Without a Data Lake: Kai Waehner | Global Field CTO | Author, International Speaker Time Series Forecasting for Managers — All Forecasts (..)
Apache Kafka 52
52 
Building a Business with a Real-Time Analytics Stack, Streaming ML Without a Data Lake, and…
ODSC - Open Data Science
MAY 24, 2023
Streaming Machine Learning Without a Data Lake The combination of data streaming and ML enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem.
Data Lakes 52
52 
Watch the Top ODSC Europe 2023 Virtual Sessions Here
ODSC - Open Data Science
JULY 14, 2023
The session participants will learn the theory behind compound sparsity, state-of-the-art techniques, and how to apply it in practice using the Neural Magic platform.
Apache Kafka 52
52 
Use streaming ingestion with Amazon SageMaker Feature Store and Amazon MSK to make ML-backed decisions in near-real time
AWS Machine Learning Blog
APRIL 19, 2023
Streaming ingestion – An Amazon Kinesis Data Analytics for Apache Flink application backed by Apache Kafka topics in Amazon Managed Streaming for Apache Kafka (MSK) (Amazon MSK) calculates aggregated features from a transaction stream, and an AWS Lambda function updates the online feature store.
ML 85
85 
Bringing Mainframe Machine and Log Data into Your ITOA Environment
Precisely
MARCH 16, 2023
Tools like Splunk, Elastic, and Apache Kafka play a central role in IT operations analytics (ITOA). Monitoring performance and security of these systems is critically important, but it does little good if you can only view that information a day or two after the fact. Today’s organizations need a real-time view of what’s happening.
Apache Kafka 52
52 
Real-time fraud detection using AWS serverless and machine learning services
AWS Machine Learning Blog
MARCH 10, 2023
The same architecture applies if you use Amazon Managed Streaming for Apache Kafka (Amazon MSK) as a data streaming service. This pattern can be useful for real-time fraud detection, notification, and potential prevention. Example use cases for this could be payment processing or high-volume account creation.
Machine Learning 122
122 
Machine Learning with MATLAB and Amazon SageMaker
NOVEMBER 21, 2023
For this particular use case, you can use streaming ingestion with Amazon SageMaker Feature Store and Amazon Managed Streaming for Apache Kafka, MSK, to make machine learning-backed decisions in near real-time.
Machine Learning 130
130 
Bundesliga Match Facts Shot Speed – Who fires the hardest shots in the Bundesliga?
AWS Machine Learning Blog
NOVEMBER 3, 2023
m How it’s implemented In our quest to accurately determine shot speed during live matches, we’ve implemented a cutting-edge solution using Amazon Managed Streaming for Apache Kafka (Amazon MSK). Example 1 Measured with top shot speed 118.43 km/h with a distance to goal of 20.61 m Example 2 Measured with top shot speed 123.32
AWS 98
98 
Unveiling Developers’ Technologies and Tools Usage in Large and Small and Medium-sized Enterprises…
Mlearning.ai
AUGUST 4, 2023
Apache Kafka and R abbitMQ are particularly popular in LEs. Graph 7: Percentage of Programming Languages MiscTech Tools In Both LEs and SMEs: ‘. NET (5+) ’, ‘ pandas ’, ‘ numpy ’, and ‘. NET Framework (1.0–4.8)’ 4.8)’ are widely used.
Apache Kafka 52
52 
Did Big Data Deliver Business Transformation & Improved CX?
Alation
AUGUST 4, 2022
Spark, Tensorflow, Apache Kafka, et cetera, are all out found in cloud databases,” points out Jones. But with the cloud, you can take a small project and test it out on new platforms with a smaller budget to start. You can] see that it works before going all-in.”.
Big Data 95
95 
Bundesliga Match Fact Ball Recovery Time: Quantifying teams’ success in pressing opponents on AWS
AWS Machine Learning Blog
MARCH 30, 2023
To ensure real-time updates of ball recovery times, we have implemented Amazon Managed Streaming for Apache Kafka (Amazon MSK) as a central solution for data streaming and messaging. This allows for seamless communication of positional data and various outputs of Bundesliga Match Facts between containers in real time.
AWS 79
79 
A Simple Guide to Real-Time Data Ingestion
Pickl AI
JULY 24, 2023
Utilising data streaming platforms such as Apache Kafka, Apache Flink, or Apache Spark Streaming, data is gathered from many sources and processed in real-time or close to real-time. IoT applications, log processing, and other data-intensive scenarios frequently use this kind of ingestion.
Apache Kafka 52
52 
Big data engineering simplified: Exploring roles of distributed systems
Data Science Dojo
JULY 24, 2023
Apache Flink for stream processing: Wrapping up In conclusion, stream processing with distributed systems like Apache Kafka, Apache Flink, and Apache Spark Streaming empowers organizations to harness real-time data insights, enabling timely decision-making and enhanced user experiences.
Let's personalize your content