Hacker News

DECEMBER 18, 2023

multi-protocol, event-native proxy. Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs. A multi-protocol, event-native proxy. Securely interface web apps, IoT clients, and microservices to Apache Kafka® via declaratively defined, stateless APIs.

Apache Kafka 98

Apache Kafka 98

IBM Journey to AI blog

MAY 9, 2023

There are thousands of business events flowing through every part of your organization every day. These events can provide a wealth of information about what’s actually happening across your business at any moment in time. But operationalizing business events is challenging.

Apache Kafka 106

Apache Kafka Business Intelligence Business Intelligence 106

IBM Journey to AI blog

JANUARY 8, 2024

In modern enterprises, where operations leave a massive digital footprint, business events allow companies to become more adaptable and able to recognize and respond to opportunities or threats as they occur. Teams want more visibility and access to events so they can reuse and innovate on the work of others.

EDA 80

EDA Apache Kafka Clustering Data Governance 80

IBM Journey to AI blog

NOVEMBER 29, 2023

By leveraging AI for real-time event processing, businesses can connect the dots between disparate events to detect and respond to new trends, threats and opportunities. AI and event processing: a two-way street An event-driven architecture is essential for accelerating the speed of business.

Artificial Intelligence 92

Artificial Intelligence Artificial Intelligence Apache Kafka AI 92

Towards AI

FEBRUARY 29, 2024

Within this article, we will explore the significance of these pipelines and utilise robust tools such as Apache Kafka and Spark to manage vast streams of data efficiently. Apache Kafka Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications.

Apache Kafka 138

Apache Kafka SQL Clustering Data Pipeline 138

Twilio Segment

SEPTEMBER 7, 2021

Event streaming platforms such as Apache Kafka are gaining in importance across all industries. In this article we'll discuss the benefits Apache Kafka implementations can gain from pairing it with a CDP.

Apache Kafka 40

Apache Kafka 40

Smart Data Collective

AUGUST 17, 2022

You can safely use an Apache Kafka cluster for seamless data movement from the on-premise hardware solution to the data lake using various cloud services like Amazon’s S3 and others. 5 Key Comparisons in Different Apache Kafka Architectures. 5 Key Comparisons in Different Apache Kafka Architectures.

Apache Kafka 110

Apache Kafka ETL Data Lakes AWS 110

IBM Journey to AI blog

NOVEMBER 21, 2023

Apache Kafka is a well-known open-source event store and stream processing platform and has grown to become the de facto standard for data streaming. Apache Kafka transfers data without validating the information in the messages. Event Streams on IBM Cloud provides a Schema Registry as part of its Enterprise plan.

Apache Kafka 107

Apache Kafka Clustering Data Quality Data Governance 107

phData

MAY 3, 2024

How Snowflake Helps Achieve Real-Time Analytics Snowflake is the ideal platform to achieve real-time analytics for several reasons, but two of the biggest are its ability to manage concurrency due to the multi-cluster architecture of Snowflake and its robust connections to 3rd party tools like Kafka. Looking for additional help?

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

ODSC - Open Data Science

FEBRUARY 6, 2024

Apache Kafka For data engineers dealing with real-time data, Apache Kafka is a game-changer. Interested in attending an ODSC event? Learn more about our upcoming events here. Each platform offers unique features and benefits, making it vital for data engineers to understand their differences.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

IBM Journey to AI blog

MARCH 14, 2024

Apache Kafka stands as a widely recognized open source event store and stream processing platform. One key advantage of opting for managed Kafka services is the delegation of responsibility for broker and operational metrics, allowing users to focus solely on metrics specific to applications. Take the next step.

Apache Kafka 93

Apache Kafka Data Pipeline 93

Precisely

DECEMBER 28, 2023

Streaming data pipelines, by extension, offer an architecture capable of handling large volumes of data, accommodating millions of events in near real time. One very popular platform is Apache Kafka , a powerful open-source tool used by thousands of companies. How do streaming data pipelines work?

Data Pipeline 91

Data Pipeline Apache Kafka Big Data Big Data 91

ODSC - Open Data Science

JUNE 1, 2023

To understand what it means, we should start by thinking of the world in terms of events, where an event is a thing that happens. And we are going to take those events, become aware of them, and understand them. Stores events in a durable manner so that downstream components can process them.

Analytics 98

Analytics Analytics Apache Kafka Data Science 98

AWS Machine Learning Blog

MARCH 10, 2023

We show how you can apply this approach to various data streaming and event-driven architectures, depending on the desired outcome and actions to take to prevent fraud (such as alert the user about the fraud or flag the transaction for additional review). The user can then choose to take action to prevent further abuse.

Machine Learning 116

Machine Learning Machine Learning AWS Apache Kafka 116

Precisely

SEPTEMBER 12, 2023

Precisely data integrity solutions fuel your Confluent and Apache Kafka streaming data pipelines with trusted data that has maximum accuracy, consistency, and context and we’re ready to share more with you at the upcoming Current 2023. Let’s cover some additional information to know before attending.

Data Silos 52

Data Silos Apache Kafka Data Pipeline Data Quality 52

ODSC - Open Data Science

JULY 22, 2023

Expo Hall ODSC events are more than just data science training and networking events. Thank you to everyone who attended for making this event possible, and showing once again why we do what we do — connecting the greater data science community together to push the industry forward. What’s next?

Apache Kafka 52

Apache Kafka Machine Learning Machine Learning Data Science 52

ODSC - Open Data Science

MAY 24, 2023

Streaming Machine Learning Without a Data Lake The combination of data streaming and ML enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem. Here’s what you can expect from ODSC Europe. Final ODSC Europe 2023 Schedule Released!

Data Lakes 52

Data Lakes ML ML Analytics 52

ODSC - Open Data Science

JULY 14, 2023

The session participants will learn the theory behind compound sparsity, state-of-the-art techniques, and how to apply it in practice using the Neural Magic platform.

Machine Learning 52

Machine Learning Machine Learning Apache Kafka Data Science 52

AWS Machine Learning Blog

NOVEMBER 3, 2023

This process comprises two key components: event data and optical tracking data. Event data collection entails gathering the fundamental building blocks of the game. For the precision needed in shot speed calculations, we must ensure that the ball’s position aligns precisely with the moment of the event.

AWS 94

AWS Apache Kafka Data Scientist Algorithm 94

ODSC - Open Data Science

MAY 31, 2023

Be sure to check out his talk, “ Apache Kafka for Real-Time Machine Learning Without a Data Lake ,” there! The combination of data streaming and machine learning (ML) enables you to build one scalable, reliable, but also simple infrastructure for all machine learning tasks using the Apache Kafka ecosystem.

Data Lakes 131

Data Lakes Machine Learning Machine Learning Apache Kafka 131

AWS Machine Learning Blog

MARCH 30, 2023

How Keeper Efficiency is implemented This Bundesliga Match Fact consumes both event and positional data. Event data consists of hand-labelled event descriptions with useful attributes, such as shot on target. This frame is used to synchronize the event data with the positional data.

Machine Learning 71

Machine Learning Machine Learning AWS Apache Kafka 71

AWS Machine Learning Blog

JANUARY 6, 2023

As the users are interacting with TR’s applications, they generate clickstream events, which are published into Amazon Kinesis Data Streams. Then the events are ingested into TR’s centralized streaming platform, which is built on top of Amazon Managed Streaming for Kafka (Amazon MSK).

AWS 67

AWS Data Warehouse ML ML 67

The MLOps Blog

FEBRUARY 5, 2023

Streaming data is a continuous flow of information and a foundation of event-driven architecture software model” – RedHat Enterprises around the world are becoming dependent on data more than ever. A streaming data pipeline is an enhanced version which is able to handle millions of events in real-time at scale.

Machine Learning 72

Machine Learning Machine Learning Data Pipeline Apache Kafka 72

Pickl AI

JULY 24, 2023

Online Gaming: Online gaming platforms require real-time data ingestion to handle large-scale events and provide a seamless experience for players. For use cases needing an ongoing data stream, like high-frequency trading or live event monitoring, continuous ingestion is a good fit.

Internet of Things 52

Internet of Things Apache Kafka ETL Azure 52

Data Science Dojo

MARCH 9, 2023

Challenges for individuals Traditional messaging brokers, such as Apache Kafka, RabbitMQ, and ActiveMQ, have been widely used to enable communication between applications and services. Additionally, there is a risk of message loss due to crashes, lack of retransmits, and poor monitoring.

Apache Kafka 326

Apache Kafka Azure Data Science Data Pipeline 326

Pickl AI

JULY 20, 2023

Diagnostic Analytics Projects: Diagnostic analytics seeks to determine the reasons behind specific events or patterns observed in the data. 3. Predictive Analytics Projects: Predictive analytics involves using historical data to predict future events or outcomes. Root cause analysis is a typical diagnostic analytics task.

Analytics 40

Analytics Analytics Big Data Big Data 40

Data Science Blog

JUNE 27, 2023

In this representation, there is a separate store for events within the speed layer and another store for data loaded during batch processing. It is important to note that in the Lambda architecture, the serving layer can be omitted, allowing batch processing and event streaming to remain separate entities.

Big Data 130

Big Data Big Data Apache Kafka Database 130

IBM Journey to AI blog

NOVEMBER 9, 2023

Apache Kafka is a high-performance, highly scalable event streaming platform. To unlock Kafka’s full potential, you need to carefully consider the design of your application. It’s all too easy to write Kafka applications that perform poorly or eventually hit a scalability brick wall.

Apache Kafka 69

Apache Kafka Algorithm Clustering 69

The MLOps Blog

MARCH 15, 2023

Data Ingestion : Involves raw data collection from origin and storage using architectures such as batch, streaming or event-driven. Pricing Up to a million events/month on the free plan. Up to 100 million events/month and a 14-day trial for the starter plan. High CPU requirement at data destination for data loading operations.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

Data Science Dojo

JULY 24, 2023

Unlike traditional batch processing, where data is processed in fixed intervals, stream processing enables organizations to gain insights and respond to events as they happen in real-time.

Big Data 195

Big Data Big Data Data Engineering Data Engineer 195

Mlearning.ai

MAY 16, 2023

In data engineering, the Pub/Sub pattern can be used for various use cases such as real-time data processing, event-driven architectures, and data synchronization across multiple systems. The company can use the Pub/Sub pattern to process customer events such as product views, add to cart, and checkout.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

AWS Machine Learning Blog

APRIL 19, 2023

Streaming ingestion – An Amazon Kinesis Data Analytics for Apache Flink application backed by Apache Kafka topics in Amazon Managed Streaming for Apache Kafka (MSK) (Amazon MSK) calculates aggregated features from a transaction stream, and an AWS Lambda function updates the online feature store.

ML 77

ML ML Apache Kafka SQL 77

Precisely

MARCH 16, 2023

Tools like Splunk, Elastic, and Apache Kafka play a central role in IT operations analytics (ITOA). Alongside a leading ITOA platform like Splunk, Elastic, or Kafka, Ironstream helps your organization improve security, lower costs, and pass compliance audits with flying colors.

Apache Kafka 52

Apache Kafka Analytics Analytics 52

DagsHub

APRIL 7, 2024

Flexibility: Airflow was designed with batch workflows in mind; it was not meant for permanently running event-based workflows. Also, while it is not a streaming solution, we can still use it for such a purpose if combined with systems such as Apache Kafka. Miscellaneous Workflows are created as directed acyclic graphs (DAGs).

Machine Learning 59

Machine Learning Machine Learning ML ML 59

The MLOps Blog

AUGUST 11, 2023

Apache Kafka, Amazon Kinesis) 2 Data Preprocessing (e.g., These include shared-nothing architecture, event-driven architecture, and directed acyclic graphs (DAGs). Today different stages exist within ML pipelines built to meet technical, industrial, and business requirements. 1 Data Ingestion (e.g.,

ML 52

ML ML Machine Learning Machine Learning 52