Data Science Blog

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. It offers full BI-Stack Automation, from source to data warehouse through to frontend.

Data Pipeline 262

Data Pipeline Data Warehouse Azure Data Lakes 262

databricks

JUNE 11, 2025

160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025.

Analytics 331

Analytics Analytics AI Data Science 331

databricks

JUNE 11, 2025

160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025.

AI 273

AI AI SQL Data Science 273

Dataconomy

JUNE 18, 2025

Data Lakehouse has emerged as a significant innovation in data management architecture, bridging the advantages of both data lakes and data warehouses. By enabling organizations to efficiently store various data types and perform analytics, it addresses many challenges faced in traditional data ecosystems.

Data Lakes 91

Data Lakes Data Warehouse Business Intelligence Business Intelligence 91

Data Science Dojo

SEPTEMBER 11, 2024

With this full-fledged solution, you don’t have to spend all your time and effort combining different services or duplicating data. OneLake, being built on Azure Data Lake Storage (ADLS), supports various data formats, including Delta, Parquet, CSV, and JSON. On the home page, select Synapse Data Engineering.

Power BI 337

Power BI Data Pipeline Data Warehouse Data Engineering 337

databricks

JUNE 18, 2025

160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025. 160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025.

AI 175

AI AI Data Science Business Intelligence 175

Data Science Connect

JANUARY 27, 2023

Data engineering is a crucial field that plays a vital role in the data pipeline of any organization. It is the process of collecting, storing, managing, and analyzing large amounts of data, and data engineers are responsible for designing and implementing the systems and infrastructure that make this possible.

Data Engineering 130

Data Engineering Data Engineering Data Engineering Data Engineer 130

databricks

JUNE 11, 2025

Product December 12, 2024 / 4 min read Making AI More Accessible: Up to 80% Cost Savings with Meta Llama 3.3 160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025.

Database 212

Database Data Lakes ETL Analytics 212

Analytics Vidhya

JANUARY 2, 2023

Introduction Delta Lake is an open-source storage layer that brings data lakes to the world of Apache Spark. Delta Lakes provides an ACID transaction–compliant and cloud–native platform on top of cloud object stores such as Amazon S3, Microsoft Azure Storage, and Google Cloud Storage.

Data Lakes 336

Data Lakes Azure Analytics Analytics 336

Pickl AI

APRIL 6, 2023

Accordingly, one of the most demanding roles is that of Azure Data Engineer Jobs that you might be interested in. The following blog will help you know about the Azure Data Engineering Job Description, salary, and certification course. How to Become an Azure Data Engineer?

Azure 52

Azure Data Engineering Data Engineering Data Engineering 52

Pickl AI

MARCH 19, 2025

Summary: Data engineering tools streamline data collection, storage, and processing. Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Learning these tools is crucial for building scalable data pipelines. Thats where data engineering tools come in!

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

databricks

JUNE 4, 2025

160 Spear Street, 15th Floor San Francisco, CA 94105 1-866-330-0121 See Careers at Databricks © Databricks 2025.

AWS 130

AWS AI AI Data Science 130

phData

FEBRUARY 21, 2025

In this post, we will be particularly interested in the impact that cloud computing left on the modern data warehouse. We will explore the different options for data warehousing and how you can leverage this information to make the right decisions for your organization. Understanding the Basics What is a Data Warehouse?

Data Warehouse 52

Data Warehouse Cloud Data ETL Cloud Computing 52

Dataconomy

JUNE 1, 2023

By automating the integration of all Fabric workloads into OneLake, Microsoft eliminates the need for developers, analysts, and business users to create their own data silos. This approach not only improves performance by eliminating the need for separate data warehouses but also results in substantial cost savings for customers.

Power BI 194

Power BI Data Lakes Azure Data Silos 194

Pickl AI

NOVEMBER 4, 2024

Summary: The fundamentals of Data Engineering encompass essential practices like data modelling, warehousing, pipelines, and integration. Understanding these concepts enables professionals to build robust systems that facilitate effective data management and insightful analysis. What is Data Engineering?

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

Mlearning.ai

JULY 8, 2023

One of them is Azure functions. In this article we’re going to check what is an Azure function and how we can employ it to create a basic extract, transform and load (ETL) pipeline with minimal code. A batch ETL works under a predefined schedule in which the data are processed at specific points in time.

ETL 52

ETL Azure Python Internet of Things 52

Pickl AI

SEPTEMBER 5, 2024

Summary: This blog provides a comprehensive roadmap for aspiring Azure Data Scientists, outlining the essential skills, certifications, and steps to build a successful career in Data Science using Microsoft Azure. What is Azure?

Azure 52

Azure Data Scientist Data Science Machine Learning 52

ODSC - Open Data Science

JANUARY 18, 2024

Data engineering is a rapidly growing field, and there is a high demand for skilled data engineers. If you are a data scientist, you may be wondering if you can transition into data engineering. In this blog post, we will discuss how you can become a data engineer if you are a data scientist.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

NOVEMBER 8, 2024

Data Versioning and Time Travel Open Table Formats empower users with time travel capabilities, allowing them to access previous dataset versions. Versioning also ensures a safer experimentation environment, where data scientists can test new models or hypotheses on historical data snapshots without impacting live data.

Data Lakes 52

Data Lakes Data Warehouse Database Azure 52

ODSC - Open Data Science

FEBRUARY 6, 2024

Data engineering has become an integral part of the modern tech landscape, driving advancements and efficiencies across industries. So let’s explore the world of open-source tools for data engineers, shedding light on how these resources are shaping the future of data handling, processing, and visualization.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

How to Learn Machine Learning

APRIL 26, 2025

The field of data science is now one of the most preferred and lucrative career options available in the area of data because of the increasing dependence on data for decision-making in businesses, which makes the demand for data science hires peak. Their insights must be in line with real-world goals.

Data Science 52

Data Science Data Analyst Data Scientist Machine Learning 52

Pickl AI

JULY 25, 2023

Unfolding the difference between data engineer, data scientist, and data analyst. Data engineers are essential professionals responsible for designing, constructing, and maintaining an organization’s data infrastructure. Read more to know.

Data Engineering 52

Data Engineering Data Engineering Data Engineering Data Engineer 52

phData

MAY 25, 2023

In a perfect world, Microsoft would have clients push even more storage and compute to its Azure Synapse platform. One of the easiest ways for Snowflake to achieve this is to have analytics solutions query their data warehouse in real-time (also known as DirectQuery).

Power BI 84

Power BI Analytics Analytics Azure 84

phData

FEBRUARY 14, 2023

Data integration is essentially the Extract and Load portion of the Extract, Load, and Transform (ELT) process. Data ingestion involves connecting your data sources, including databases, flat files, streaming data, etc, to your data warehouse. Snowflake provides native ways for data ingestion.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

ODSC - Open Data Science

JUNE 7, 2023

Data has to be stored somewhere. Data warehouses are repositories for your cleaned, processed data, but what about all that unstructured data your organization is starting to notice? What is a data lake? So let’s take a look at a few of the leading industry examples of data lakes. Where does it go?

Data Lakes 52

Data Lakes Azure Data Warehouse Hadoop 52

Mlearning.ai

FEBRUARY 16, 2023

The ultimate need for vast storage spaces manifests in data warehouses: specialized systems that aggregate data coming from numerous sources for centralized management and consistency. In this article, you’ll discover what a Snowflake data warehouse is, its pros and cons, and how to employ it efficiently.

Data Warehouse 52

Data Warehouse Business Intelligence Business Intelligence Database 52

phData

SEPTEMBER 2, 2024

Best practices are a pivotal part of any software development, and data engineering is no exception. This ensures the data pipelines we create are robust, durable, and secure, providing the desired data to the organization effectively and consistently. Below are the best practices.

ETL 52

ETL Data Warehouse SQL Database 52

Pickl AI

OCTOBER 17, 2024

Introduction ETL plays a crucial role in Data Management. This process enables organisations to gather data from various sources, transform it into a usable format, and load it into data warehouses or databases for analysis. Loading The transformed data is loaded into the target destination, such as a data warehouse.

ETL 52

ETL Data Warehouse Data Quality Data Governance 52

Pickl AI

APRIL 9, 2025

By 2025, global data volumes are expected to reach 181 zettabytes, according to IDC. To harness this data effectively, businesses rely on ETL (Extract, Transform, Load) tools to extract, transform, and load data into centralized systems like data warehouses. Cost : Is the pricing predictable and within budget?

ETL 52

ETL Data Warehouse AWS Business Intelligence 52

Analytics Vidhya

FEBRUARY 25, 2023

Introduction A data lake is a centralized and scalable repository storing structured and unstructured data. The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.

Data Lakes 178

Data Lakes Analytics Analytics Data Warehouse 178

phData

SEPTEMBER 19, 2023

Thankfully, there are tools available to help with metadata management, such as AWS Glue, Azure Data Catalog, or Alation, that can automate much of the process. What are the Best Data Modeling Methodologies and Processes? Data lakes are meant to be flexible for new incoming data, whether structured or unstructured.

Data Lakes 52

Data Lakes Data Models Data Modeling Data Warehouse 52

ODSC - Open Data Science

OCTOBER 7, 2024

There you’ll hear from Ivan Nardini, Developer Relations Engineer at Google Cloud and discover the latest advancements in AI and learn how to leverage Google Cloud’s powerful tools and infrastructure to drive innovation in your organization.

Data Scientist 52

Data Scientist Data Science Machine Learning Machine Learning 52

phData

NOVEMBER 28, 2023

Matillion is also built for scalability and future data demands, with support for cloud data platforms such as Snowflake Data Cloud , Databricks, Amazon Redshift, Microsoft Azure Synapse, and Google BigQuery, making it future-ready, everyone-ready, and AI-ready. That process will not take longer than 3 minutes!

Data Warehouse 52

Data Warehouse Data Pipeline ETL Azure 52

Pickl AI

JUNE 7, 2024

Also Read: Top 10 Data Science tools for 2024. It is a process for moving and managing data from various sources to a central data warehouse. This process ensures that data is accurate, consistent, and usable for analysis and reporting. This process helps organisations manage large volumes of data efficiently.

ETL 40

ETL Data Quality Data Pipeline Data Warehouse 40

phData

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. First up, let’s dive into the foundation of every Modern Data Stack, a cloud-based data warehouse.

Data Warehouse 52

Data Warehouse Analytics Analytics Cloud Data 52

phData

AUGUST 9, 2024

Typically, this data is scattered across Excel files on business users’ desktops. Cloud Storage Upload Snowflake can easily upload files from cloud storage (AWS S3, Azure Storage, GCP Cloud Storage). Hosting the data in Snowflake’s hybrid tables may be a better choice for these applications.

ETL 52

ETL Data Warehouse Data Governance Tableau 52

The MLOps Blog

MAY 17, 2023

This article explores the importance of ETL pipelines in machine learning, a hands-on example of building ETL pipelines with a popular tool, and suggests the best ways for data engineers to enhance and sustain their pipelines. It comprises three main areas: Landing area, Staging area, and Data Warehouse area.

ETL 59

ETL Data Pipeline ML ML 59