Dataversity

MAY 26, 2023

Suppose you’re in charge of maintaining a large set of data pipelines from cloud storage or streaming data into a data warehouse. How can you ensure that your data meets expectations after every transformation? That’s where data quality testing comes in.

Data Pipeline 52

Data Pipeline Data Warehouse Data Quality Data Observability 52

Dataversity

JUNE 19, 2023

In part one of this article, we discussed how data testing can specifically test a data object (e.g., table, column, metadata) at one particular point in the data pipeline.

Data Pipeline 52

Data Pipeline Database Data Modeling Data Models 52

Alation

FEBRUARY 20, 2020

The rise of data lakes, IOT analytics, and big data pipelines has introduced a new world of fast, big data. How Data Catalogs Can Help. Data catalogs evolved as a key component of the data governance revolution by creating a bridge between the new world and old world of data governance.

Data Governance 52

Data Governance Tableau Analytics Analytics 52

Alation

FEBRUARY 17, 2023

This past week, I had the pleasure of hosting Data Governance for Dummies author Jonathan Reichental for a fireside chat , along with Denise Swanson , Data Governance lead at Alation. Can you have proper data management without establishing a formal data governance program?

Data Governance 52

Data Governance Data Quality Data Analyst Data Pipeline 52

Alation

JANUARY 26, 2023

That’s why many organizations invest in technology to improve data processes, such as a machine learning data pipeline. However, data needs to be easily accessible, usable, and secure to be useful — yet the opposite is too often the case. These data requirements could be satisfied with a strong data governance strategy.

Data Governance 52

Data Governance Data Engineering Data Engineer Data Engineering 52

IBM Journey to AI blog

FEBRUARY 26, 2024

The financial services industry has been in the process of modernizing its data governance for more than a decade. But as we inch closer to global economic downturn, the need for top-notch governance has become increasingly urgent. That’s why data pipeline observability is so important.

Data Pipeline 112

Data Pipeline Data Engineering Data Engineering Data Engineering 112

Dataversity

JUNE 26, 2023

The same expectation applies to data, […] The post Leveraging Data Pipelines to Meet the Needs of the Business: Why the Speed of Data Matters appeared first on DATAVERSITY. Today, businesses and individuals expect instant access to information and swift delivery of services.

Data Pipeline 52

Data Pipeline Data Observability Data Quality Data Governance 52

Dataversity

MARCH 13, 2023

Companies are spending a lot of money on data and analytics capabilities, creating more and more data products for people inside and outside the company. These products rely on a tangle of data pipelines, each a choreography of software executions transporting data from one place to another.

Data Observability 98

Data Observability Data Pipeline Analytics Analytics 98

IBM Journey to AI blog

APRIL 10, 2023

To further the above, organizations should have the right foundation that consists of a modern data governance approach and data architecture. It’s becoming critical that organizations should adopt a data architecture that supports AI governance. The time for data professionals to meet this challenge is now.

AI 103

AI AI Data Governance Data Pipeline 103

IBM Journey to AI blog

APRIL 20, 2023

This trust depends on an understanding of the data that inform risk models: where does it come from, where is it being used, and what are the ripple effects of a change? Moreover, banks must stay in compliance with industry regulations like BCBS 239, which focus on improving banks’ risk data aggregation and risk reporting capabilities.

Database 86

Database Data Engineering Data Engineering Data Engineering 86

Alation

FEBRUARY 20, 2020

Alation’s deep integration with tools like MicroStrategy and Tableau provides visibility into the complete data pipeline: from storage through visualization. Subscribe to Alation's Blog. Get the latest data cataloging news and trends in your inbox. The post Cataloging MicroStrategy appeared first on Alation.

Data Governance 52

Data Governance Tableau Hadoop Data Pipeline 52

Pickl AI

OCTOBER 11, 2023

Data Observability and Data Quality are two key aspects of data management. The focus of this blog is going to be on Data Observability tools and their key framework. The growing landscape of technology has motivated organizations to adopt newer ways to harness the power of data. What is Data Observability?

Data Observability 52

Data Observability Data Quality Data Pipeline Data Governance 52

IBM Journey to AI blog

OCTOBER 12, 2023

Securing AI models and their access to data While AI models need flexibility to access data across a hybrid infrastructure, they also need safeguarding from tampering (unintentional or otherwise) and, especially, protected access to data. And that makes sense.

AI 93

AI AI Data Scientist Data Governance 93

Pickl AI

OCTOBER 10, 2023

In this blog, we are going to unfold the two key aspects of data management that is Data Observability and Data Quality. Data is the lifeblood of the digital age. Today, every organization tries to explore the significant aspects of data and its applications.

Data Observability 52

Data Observability Data Quality Data Governance Data Pipeline 52

Alation

SEPTEMBER 22, 2022

This new partnership will unify governed, quality data into a single view, granting all stakeholders total visibility into pipelines and providing them with a superior ability to make data-driven decisions. For people to understand and trust data, they need to see it in context. Data Pipeline Strategy.

Data Pipeline 52

Data Pipeline Data Quality Data Governance Data Engineering 52

AWS Machine Learning Blog

DECEMBER 18, 2024

All data generation and processing steps were run in parallel directly on the SageMaker HyperPod cluster nodes, using a unique working environment and highlighting the clusters versatility for various tasks beyond just training models. She specializes in AI operations, data governance, and cloud architecture on AWS.

Clustering 103

Clustering AWS AI AI 103

IBM Journey to AI blog

JANUARY 17, 2023

IBM Cloud Pak for Data Express solutions provide new clients with affordable and high impact capabilities to expeditiously explore and validate the path to become a data-driven enterprise. IBM Cloud Pak for Data Express solutions offer clients a simple on ramp to start realizing the business value of a modern architecture.

Data Governance 58

Data Governance Data Science AI AI 58

IBM Journey to AI blog

JANUARY 9, 2024

The groundwork of training data in an AI model is comparable to piloting an airplane. The entire generative AI pipeline hinges on the data pipelines that empower it, making it imperative to take the correct precautions. This may also entail working with new data through methods like web scraping or uploading.

AI 89

AI AI Data Quality Data Pipeline 89

phData

AUGUST 5, 2024

How can my analysts discover where data is located? All of these questions describe a concept known as data governance. The Snowflake AI Data Cloud has built an entire blanket of features called Horizon, which tackles all of these questions and more. We will begin with compliance. appeared first on phData.

Data Governance 52

Data Governance Data Quality Data Lakes ML 52

Alation

DECEMBER 23, 2021

The best data was discovered, experts were identified, and conversations were starting. For the first time, data governance was no longer a naughty concept. Yup, the big syndicate was doing data culture – nice data culture. Now, elves of all rank and file can: Know their data and how they can use it.

Data Governance 59

Data Governance Data Pipeline Tableau Big Data 59

Alation

DECEMBER 7, 2021

Do we have end-to-end data pipeline control? What can we learn about our data quality issues? How can we improve and deliver trusted data to the organization? One major obstacle presented to data quality is data silos , as they obstruct transparency and make collaboration tough. Subscribe to Alation's Blog.

Data Quality 52

Data Quality Data Pipeline Data Silos Data Governance 52

Alation

APRIL 19, 2022

Modern data architectures, like cloud data warehouses and cloud data lakes , empower more people to leverage analytics for insights more efficiently. Access the resources your data applications need — no more, no less. Data Pipeline Automation. What Is the Role of Data Governance in Data Modernization?

Data Governance 52

Data Governance Cloud Data Database Data Silos 52

IBM Journey to AI blog

AUGUST 12, 2024

It helps companies streamline and automate the end-to-end ML lifecycle, which includes data collection, model creation (built on data sources from the software development lifecycle), model deployment, model orchestration, health monitoring and data governance processes.

Big Data 106

Big Data Big Data ML ML 106

phData

APRIL 23, 2024

Data teams are now tasked with designing and maintaining scaleable, flexible data architecture to support a wide variety of business-critical data-driven reports and analytics. Engineering teams must maintain a complex web of ingestion pipelines capable of supporting many different sources, each with its own intricacies.

Data Governance 52

Data Governance Data Pipeline Data Warehouse Cloud Data 52

phData

APRIL 29, 2024

In our previous blog, Top 5 Fivetran Connectors for Financial Services , we explored Fivetran’s capabilities that address the data integration needs of the finance industry. Now, let’s cover the healthcare industry, which also has a surging demand for data and analytics, along with the underlying processes to make it happen.

SQL 52

SQL Data Warehouse Azure Cloud Data 52

Alation

APRIL 18, 2023

For any data user in an enterprise today, data profiling is a key tool for resolving data quality issues and building new data solutions. In this blog, we’ll cover the definition of data profiling, top use cases, and share important techniques and best practices for data profiling today.

Data Profiling 52

Data Profiling Data Quality Data Governance Data Pipeline 52

The MLOps Blog

JUNE 27, 2023

It sits between the data lake and cloud object storage, allowing you to version and control changes to data lakes at scale. LakeFS facilitates data reproducibility, collaboration, and data governance within the data lake environment. Flyte Flyte is a platform for orchestrating ML pipelines at scale.

Machine Learning 134

Machine Learning Machine Learning ML ML 134

Flipboard

MAY 30, 2025

Data governance Apply fine-grained access control to data managed by the system, including training data, vector stores, evaluation data, prompt templates, workflow, and agent definitions. Model versions should be managed centrally in a model registry. Access controls to models should be established.

AWS 115

AWS AI AI Database 115

IBM Journey to AI blog

AUGUST 4, 2023

Data democratization instead refers to the simplification of all processes related to data, from storage architecture to data management to data security. It also requires an organization-wide data governance approach, from adopting new types of employee training to creating new policies for data storage.

Data Lakes 73

Data Lakes AI AI Data Governance 73

Alation

AUGUST 3, 2021

DataOps then works to continuously improve and adjust data models, visualizations, reports, and dashboards to achieve business goals. DataOps fosters cross-functional collaboration and automation to build fast, trustworthy data pipelines so your business can wring the most value from your data. The Agile Connection.

DataOps 59

DataOps Data Pipeline Data Analyst Analytics 59

Alation

DECEMBER 7, 2021

Data teams use Bigeye’s data observability platform to detect data quality issues and ensure reliable data pipelines. If there is an issue with the data or data pipeline, the data team is immediately alerted, enabling them to proactively address the issue. Subscribe to Alation's Blog.

Data Quality 52

Data Quality Data Pipeline Data Observability Data Profiling 52

Pickl AI

JULY 25, 2023

Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. They create data pipelines, ETL processes, and databases to facilitate smooth data flow and storage. Read more to know. Cloud Platforms: AWS, Azure, Google Cloud, etc.

Data Engineering 52

Data Engineering Data Engineer Data Engineering Data Engineering 52

Alation

APRIL 14, 2022

Semantics, context, and how data is tracked and used mean even more as you stretch to reach post-migration goals. This is why, when data moves, it’s imperative for organizations to prioritize data discovery. Data discovery is also critical for data governance , which, when ineffective, can actually hinder organizational growth.

Data Governance 59

Data Governance ML ML Cloud Data 59

phData

AUGUST 2, 2024

Snowflake AI Data Cloud is one of the most powerful platforms, including storage services supporting complex data. Integrating Snowflake with dbt adds another layer of automation and control to the data pipeline. In this blog, we’ll explore: Overview of Snowflake Stored Procedures & dbt Hooks.

Data Pipeline 52

Data Pipeline Python Database SQL 52

IBM Journey to AI blog

JULY 13, 2023

This is the practice of creating, updating and consistently enforcing the processes, rules and standards that prevent errors, data loss, data corruption, mishandling of sensitive or regulated data, and data breaches. Learn more about designing the right data architecture to elevate your data quality here.

Data Quality 74

Data Quality Data Profiling Data Governance Machine Learning 74

AWS Machine Learning Blog

FEBRUARY 13, 2024

Let’s demystify this using the following personas and a real-world analogy: Data and ML engineers (owners and producers) – They lay the groundwork by feeding data into the feature store Data scientists (consumers) – They extract and utilize this data to craft their models Data engineers serve as architects sketching the initial blueprint.

AWS 134

AWS ML ML Machine Learning 134

phData

JUNE 20, 2024

It is the ideal single source of truth to support analytics and drive data adoption – the foundation of the data culture! In this blog, we’ll walk you through how to build a sustainable data culture with Snowflake. Understanding Data Culture A data culture is really about people having trust in the data.

Data Governance 52

Data Governance Analytics Analytics Power BI 52