ODSC - Open Data Science

DECEMBER 5, 2023

You can see our photos from the event here , and be sure to follow our YouTube for virtual highlights from the conference as well. Over in San Francisco, we had a keynote for each day of the event. Other Events Aside from networking events and all of our sessions, we had a few other special events. What’s next?

Data Science 98

Data Science Artificial Intelligence Artificial Intelligence Machine Learning 98

The MLOps Blog

MARCH 15, 2023

In this post, you will learn about the 10 best data pipeline tools, their pros, cons, and pricing. A typical data pipeline involves the following steps or processes through which the data passes before being consumed by a downstream process, such as an ML model training process.

Data Pipeline 59

Data Pipeline ETL SQL Data Quality 59

IBM Journey to AI blog

MARCH 14, 2024

Apache Kafka stands as a widely recognized open source event store and stream processing platform. It has evolved into the de facto standard for data streaming, as over 80% of Fortune 500 companies use it. All major cloud providers provide managed data streaming services to meet this growing demand. Take the next step.

Apache Kafka 91

Apache Kafka Data Pipeline 91

IBM Journey to AI blog

NOVEMBER 3, 2023

Apache Kafka and Apache Flink working together Anyone who is familiar with the stream processing ecosystem is familiar with Apache Kafka: the de-facto enterprise standard for open-source event streaming. Apache Kafka streams get data to where it needs to go, but these capabilities are not maximized when Apache Kafka is deployed in isolation.

Apache Kafka 75

Apache Kafka Data Warehouse Data Pipeline SQL 75

AWS Machine Learning Blog

FEBRUARY 24, 2023

Solution overview In brief, the solution involved building three pipelines: Data pipeline – Extracts the metadata of the images Machine learning pipeline – Classifies and labels images Human-in-the-loop review pipeline – Uses a human team to review results The following diagram illustrates the solution architecture.

AWS 86

AWS Data Pipeline ML ML 86

phData

APRIL 24, 2024

This article was co-written by Mayank Singh & Ayush Kumar Singh Your organization’s data pipelines will inevitably run into issues, ranging from simple permission errors to significant network or infrastructure incidents. Failed Webhooks If webhooks are configured and the webhook event fails, a notification will be sent out.

Data Pipeline 52

Data Pipeline ETL Azure Analytics 52

ODSC - Open Data Science

FEBRUARY 6, 2024

Apache Kafka For data engineers dealing with real-time data, Apache Kafka is a game-changer. This open-source streaming platform enables the handling of high-throughput data feeds, ensuring that data pipelines are efficient, reliable, and capable of handling massive volumes of data in real-time.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

OCTOBER 15, 2023

Elementl / Dagster Labs Elementl and Dagster Labs are both companies that provide platforms for building and managing data pipelines. Elementl’s platform is designed for data engineers, while Dagster Labs’ platform is designed for data scientists. Interested in attending an ODSC event?

Machine Learning 52

Machine Learning Machine Learning Data Pipeline AI 52

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data. Let’s cover some additional information to know before attending.

Data Silos 52

Data Silos Apache Kafka Data Pipeline Data Quality 52

ODSC - Open Data Science

DECEMBER 19, 2023

Data Engineer Data engineers are responsible for the end-to-end process of collecting, storing, and processing data. They use their knowledge of data warehousing, data lakes, and big data technologies to build and maintain data pipelines. Interested in attending an ODSC event?

Data Scientist 96

Data Scientist Machine Learning Machine Learning AI 96

Precisely

MAY 13, 2024

You should also be able to deploy data pipelines anywhere your data lives, whether it be on-premises, public cloud, private cloud, multi-cloud, or hybrid environment. Questions to ask each vendor: How do your solutions provide data delivery resiliency and backup in the event of a service disruption?

Data Governance 52

Data Governance Data Pipeline Data Quality Cloud Data 52

Dataversity

APRIL 30, 2021

The goal of digital transformation remains the same as ever – to become more data-driven. We have learned how to gain a competitive advantage by capturing business events in data. Events are data snap-shots of complex activity sourced from the web, customer systems, ERP transactions, social media, […].

Data Pipeline 98

Data Pipeline Data Warehouse Data Governance Data Quality 98

ODSC - Open Data Science

MAY 9, 2024

Whether you’re managing data pipelines or deploying machine learning models, Thunder makes the process smooth and efficient. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Interested in attending an ODSC event? Learn more about our upcoming events here.

AI 52

AI AI Machine Learning Machine Learning 52

Alation

APRIL 14, 2023

People were familiar with the value of a data catalog (and the growing need for data governance ), though many admitted to being somewhat behind on their journeys. In this blog, I’ll share a quick high-level overview of the event, with an eye to core themes. What did attendees take away from the event? Let’s dive in!

Data Pipeline 52

Data Pipeline Data Warehouse Cloud Data ETL 52

ODSC - Open Data Science

JANUARY 18, 2024

Data engineers will also work with data scientists to design and implement data pipelines; ensuring steady flows and minimal issues for data teams. They’ll also work with software engineers to ensure that the data infrastructure is scalable and reliable. Interested in attending an ODSC event?

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

The MLOps Blog

FEBRUARY 5, 2023

In the later part of this article, we will discuss its importance and how we can use machine learning for streaming data analysis with the help of a hands-on example. What is streaming data? This will also help us observe the importance of stream data. It can be used to collect, store, and process streaming data in real-time.

Machine Learning 72

Machine Learning Machine Learning Data Pipeline Apache Kafka 72

Flipboard

SEPTEMBER 25, 2023

Brian Chesky, CEO of Airbnb, spoke at a Y Combinator event this summer. (Y Neum AI Photo) Co-founders: David de Matheu and Pinhas Kevin Cohen Explain what your startup does in two sentences: Neum AI is the next generation of data pipelines built specifically for retrieval augmented generation (RAG).

Data Pipeline 137

Data Pipeline AI AI Natural Language Processing 137

Flipboard

APRIL 13, 2023

“The amount of the data that we process every day and make available for researchers in a timely fashion makes it a very complex and really big data problem,” said Jay Nanduri , Truveta chief technology officer, in an interview with GeekWire. Last fall, Truveta also unveiled Truveta Studio , an interface into real-time patient data.

AI 157

AI AI Data Pipeline Big Data 157

Towards AI

FEBRUARY 29, 2024

Apache Kafka plays a crucial role in enabling data processing in real-time by efficiently managing data streams and facilitating seamless communication between various components of the system. Apache Kafka Apache Kafka is a distributed event streaming platform used for building real-time data pipelines and streaming applications.

Apache Kafka 134

Apache Kafka SQL Clustering Data Pipeline 134

phData

JUNE 21, 2023

Operational Risks: Uncover operational risks such as data loss or failures in the event of an unforeseen outage or disaster. Performance Optimization: Locate and fix bottlenecks in your data pipelines so that you can get the most out of your Snowflake investment.

Data Pipeline 52

Data Pipeline 52

Heartbeat

NOVEMBER 6, 2023

Image Source —  Pixel Production Inc In the previous article, you were introduced to the intricacies of data pipelines, including the two major types of existing data pipelines. You might be curious how a simple tool like Apache Airflow can be powerful for managing complex data pipelines.

Data Pipeline 52

Data Pipeline Clean Data ETL Python 52

Precisely

FEBRUARY 2, 2023

Business managers are faced with plotting the optimal course in the face of these evolving events. Pipelines must have robust data integration capabilities that integrate data from multiple data silos, including the extensive list of applications used throughout the organization, databases and even mainframes.

Data Pipeline 52

Data Pipeline Data Silos Database Data Observability 52

phData

FEBRUARY 3, 2023

This provides an adequate “pipeline” to load a lot of fresh data in the form of micro-batches. With Snowpipe’s feature of automated data loading, it also leverages event notification for the purpose of cloud storage. This functionality also informs Snowpipe of the arrival of the new data for loading.

ETL 59

ETL Data Warehouse Data Pipeline Analytics 59

ODSC - Open Data Science

SEPTEMBER 29, 2023

Furthermore, we’ve developed data encryption and governance solutions for HPCC Systems to help secure data, ensure it is only accessed by appropriate personnel, and to create audit trails to ensure data security SLAs and regulations are met. It truly is an all-in-one data lake solution.

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

ODSC - Open Data Science

APRIL 4, 2024

Find out how to weave data reliability and quality checks into the execution of your data pipelines and more. Announcing the ODSC East 2024 Complete Schedule Check out this article for a full rundown of what you can expect from the ODSC East 2024 schedule, including keynotes, bootcamp sessions, extra events, and more.

Data Visualization 52

Data Visualization Analytics Analytics Big Data Analytics 52

phData

MAY 3, 2024

What is Apache Kafka, and How is it Used in Building Real-time Data Pipelines? Apache Kafka is an open-source event distribution platform. It is capable of handling high-volume and high-velocity data. It is highly scalable, has high availability, and has low latency.

Apache Kafka 52

Apache Kafka Analytics Analytics ETL 52

Smart Data Collective

AUGUST 17, 2022

Kafka And ETL Processing: You might be using Apache Kafka for high-performance data pipelines, stream various analytics data, or run company critical assets using Kafka, but did you know that you can also use Kafka clusters to move data between multiple systems. 5 Key Comparisons in Different Apache Kafka Architectures.

Apache Kafka 110

Apache Kafka ETL Data Lakes AWS 110

Heartbeat

JANUARY 5, 2024

Introduction to LangChain for Including AI from Large Language Models (LLMs) Inside Data Applications and Data Pipelines This article will provide an overview of LangChain, the problems it addresses, its use cases, and some of its limitations. Python : Great for including AI in Python-based software or data pipelines.

AI 52

AI AI Data Pipeline Deep Learning 52

AWS Machine Learning Blog

OCTOBER 18, 2023

AWS Lambda is an event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. Tayo Olajide is a seasoned Cloud Data Engineering generalist with over a decade of experience in architecting and implementing data solutions in cloud environments.

AWS 96

AWS ML ML Machine Learning 96

Smart Data Collective

SEPTEMBER 19, 2022

Historical financial big data helps businesses scrutinize evolving customer behaviors, allowing them to come up with invaluable products and services that streamline banking processes. However, to take full advantage of big data’s powerful capabilities, choosing BI and ETL solutions cannot be over-emphasized.

Big Data 129

Big Data Big Data Big Data Analytics Big Data Analytics 129

phData

JANUARY 24, 2024

In addition to the Application type, Fivetran provides connectors for databases, files, events, and functions. Salesforce – The Salesforce (SFDC) connector streamlines customer relationship management (CRM) data integration. Scalability Considerations: Ensure your data integration setup is scalable and ready for future growth.

Data Warehouse 52

Data Warehouse Data Pipeline Data Governance Data Quality 52

Snorkel AI

JUNE 9, 2023

Snorkel AI wrapped the second day of our The Future of Data-Centric AI virtual conference by showcasing how Snorkel’s data-centric platform has enabled customers to succeed, taking a deep look at Snorkel Flow’s capabilities, and announcing two new solutions. You need to find a place to park your data.

AI 52

AI AI Data Scientist Machine Learning 52

Snorkel AI

JUNE 9, 2023

Snorkel AI wrapped the second day of our The Future of Data-Centric AI virtual conference by showcasing how Snorkel’s data-centric platform has enabled customers to succeed, taking a deep look at Snorkel Flow’s capabilities, and announcing two new solutions. You need to find a place to park your data.

AI 52

AI AI Data Scientist Machine Learning 52

phData

MARCH 29, 2024

Operational Risks: Uncover operational risks such as data loss or failures in the event of an unforeseen outage or disaster. Performance Optimization: Locate and fix bottlenecks in your data pipelines so that you can get the most out of your Snowflake investment.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

ODSC - Open Data Science

JUNE 19, 2023

As the name suggests, real-time operating systems (RTOS) handle real-time applications that undertake data and event processing under a strict deadline. RTOS takes away most of the OS-associated obstacles or challenges in advancing big data processing.

Big Data 96

Big Data Big Data Artificial Intelligence Artificial Intelligence 96

IBM Journey to AI blog

SEPTEMBER 19, 2023

By analyzing datasets, data scientists can better understand their potential use in an algorithm or machine learning model. The data science lifecycle Data science is iterative, meaning data scientists form hypotheses and experiment to see if a desired outcome can be achieved using available data.

Data Science 83

Data Science Analytics Analytics Data Scientist 83

Heartbeat

JANUARY 3, 2024

Our planet sends distress signals through extreme weather events, melting ice caps, and vanishing species. This real-time capability is critical for managing fast-evolving situations, such as extreme weather events or environmental emergencies, where timely intervention can mitigate risks and minimize impacts.

Machine Learning 52

Machine Learning Machine Learning ML ML 52