Hacker News

JULY 18, 2024

ABOUT EVENTUAL Eventual is a data platform that helps data scientists and engineers build data applications across ETL, analytics and ML/AI. OUR PRODUCT IS OPEN-SOURCE AND USED AT ENTERPRISE SCALE Our distributed data engine Daft [link] is open-sourced and runs on 800k CPU cores daily. WE'RE GROWING - COME GROW WITH US!

ML 126

ML ML Python ETL 126

Hacker News

JUNE 9, 2025

Python: https://github.com/chonkie-inc/chonkie TypeScript: https://github.com/chonkie-inc/chonkie-ts Here's a video showing our code chunker: https://youtu.be/Xclkh6bU1P0. . Is Chonkie primarily for people looking to process documents in some sort of real-time scenario?

Database 81

Database SQL ETL AI 81

IBM Journey to AI blog

OCTOBER 30, 2024

This solution supports the validation of adherence to existing obligations by analyzing governance documents and controls in place and mapping them to applicable LRRs. This approach enables centralized access and sharing while minimizing extract, transform and load (ETL) processes and data duplication. Furthermore, watsonx.ai

Machine Learning 95

Machine Learning Machine Learning ETL AI 95

Pickl AI

JUNE 7, 2024

Summary: Choosing the right ETL tool is crucial for seamless data integration. At the heart of this process lie ETL Tools—Extract, Transform, Load—a trio that extracts data, tweaks it, and loads it into a destination. Choosing the right ETL tool is crucial for smooth data management. What is ETL?

ETL 40

ETL Data Quality Data Pipeline Data Warehouse 40

Flipboard

NOVEMBER 24, 2023

This use case highlights how large language models (LLMs) are able to become a translator between human languages (English, Spanish, Arabic, and more) and machine interpretable languages (Python, Java, Scala, SQL, and so on) along with sophisticated internal reasoning.

Database 156

Database AWS ETL SQL 156

AWS Machine Learning Blog

FEBRUARY 18, 2025

An Amazon EventBridge schedule checked this bucket hourly for new files and triggered log transformation extract, transform, and load (ETL) pipelines built using AWS Glue and Apache Spark. Creating ETL pipelines to transform log data Preparing your data to provide quality results is the first step in an AI project.

AWS 73

AWS Database ETL AI 73

phData

APRIL 28, 2025

A Matillion pipeline is a collection of jobs that extract, load, and transform (ETL/ELT) data from various sources into a target system, such as a cloud data warehouse like Snowflake. Intuitive Workflow Design Workflows should be easy to follow and visually organized, much like clean, well-structured SQL or Python code.

AI 52

AI AI SQL ETL 52

Applied Data Science

MAY 6, 2021

To keep myself sane, I use Airflow to automate tasks with simple, reusable pieces of code for frequently repeated elements of projects, for example: Web scraping ETL Database management Feature building and data validation And much more! Take a quick look at the architecture diagram below, from the Airflow documentation.

Data Scientist 130

Data Scientist Python Data Science Database 130

AWS Machine Learning Blog

FEBRUARY 21, 2025

The agent knowledge base stores Amazon Bedrock service documentation, while the cache knowledge base contains curated and verified question-answer pairs. This setup uses the AWS SDK for Python (Boto3) to interact with AWS services. Step 1: Set up two Amazon Bedrock knowledge bases This step creates two Amazon Bedrock knowledge bases.

AWS 127

AWS Natural Language Processing Machine Learning Machine Learning 127

Towards AI

MARCH 21, 2023

PowerBI, Tableau) and programming languages like R and Python in the form of bar graphs, scatter line plots, histograms, and much more. What are ETL and data pipelines? The source of extraction of data can be files like text files, excel sheets, word documents, databases like relational as well as non-relational, and also the APIs.

Data Engineering 98

Data Engineering Data Engineering Data Engineer Data Engineering 98

AWS Machine Learning Blog

MAY 5, 2023

The following figure shows an example diagram that illustrates an orchestrated extract, transform, and load (ETL) architecture solution. Using architecture diagrams as an example, the solution needs to search through reference links and technical documents for architecture diagrams and identify the services present.

AWS 104

AWS ETL ML ML 104

Smart Data Collective

APRIL 29, 2020

Extraction, Transform, Load (ETL). Dataform enables the creation of a central repository for defining data throughout an organisation, as well as discovering datasets and documenting data in a catalogue. It allows users to organise, monitor and schedule ETL processes through the use of Python. Master data management.

Data Warehouse 99

Data Warehouse Azure SQL ETL 99

AWS Machine Learning Blog

NOVEMBER 29, 2023

You can use this notebook job step to easily run notebooks as jobs with just a few lines of code using the Amazon SageMaker Python SDK. These jobs can be run immediately or on a recurring time schedule without the need for data workers to refactor code as Python modules. Refer to SageMaker documentation for detailed instructions.

ML 117

ML ML Data Scientist Python 117

Pickl AI

MARCH 19, 2025

Tools like Python, SQL, Apache Spark, and Snowflake help engineers automate workflows and improve efficiency. Python, SQL, and Apache Spark are essential for data engineering workflows. Python Python is one of the most popular programming languages for data engineering. Start your journey with Pickl.AI

Data Engineering 52

Data Engineering Data Engineering Data Engineer Data Engineering 52

AWS Machine Learning Blog

SEPTEMBER 18, 2024

Airflow for workflow orchestration Airflow schedules and manages complex workflows, defining tasks and dependencies in Python code. Though it’s worth mentioning that Airflow isn’t used at runtime as is usual for extract, transform, and load (ETL) tasks. Every Airflow task calls Amazon ECS tasks with some overrides.

AWS 123

AWS Machine Learning Machine Learning ML 123

AWS Machine Learning Blog

JUNE 25, 2024

When the automated content processing steps are complete, you can use the output for downstream tasks, such as to invoke different components in a customer service backend application, or to insert the generated tags into metadata of each document for product recommendation.

AWS 128

AWS Natural Language Processing Machine Learning Machine Learning 128

phData

AUGUST 9, 2024

They usually operate outside any data governance structure; often, no documentation exists outside the user’s mind. Host in SharePoint or Google Docs A simple and common option is to leave the data in a spreadsheet but host it in a document management service. This allows for easy sharing and collaboration on the data.

ETL 52

ETL Data Warehouse Data Governance Tableau 52

The MLOps Blog

MARCH 15, 2023

Documentation: Keep detailed documentation of the deployed model, including its architecture, training data, and performance metrics, so that it can be understood and managed effectively. If you aren’t aware already, let’s introduce the concept of ETL. We primarily used ETL services offered by AWS.

AWS 52

AWS ETL ML ML 52

ODSC - Open Data Science

OCTOBER 18, 2024

Audit existing data assets Inventory internal datasets, ETL capabilities, past analytical initiatives, and available skill sets. Usability Do interfaces and documentation enable business analysts and data scientists to leverage systems? Prioritize libraries with strong community support like Python and R.

Data Science 52

Data Science Data Scientist ETL Analytics 52

phData

AUGUST 8, 2024

Putting the T for Transformation in ELT (ETL) is essential to any data pipeline. In Snowflake, stored procedures can be created in normal SQL and in Javascript, Python, Java, and Scala (the latter three need to be made using the Snowpark API). Coalesce’s top features include column-level lineage and auto-generated documentation.

SQL 52

SQL Data Pipeline Python ETL 52

Heartbeat

NOVEMBER 6, 2023

You also learned how to build an Extract Transform Load (ETL) pipeline and discovered the automation capabilities of Apache Airflow for ETL pipelines. This is necessary because additional Python modules need to be installed. Similarly, if you need to complete a Python process, you will need the Python operator.

Data Pipeline 52

Data Pipeline Clean Data ETL Python 52

Alation

JANUARY 17, 2023

Reverse ETL tools. The modern data stack is also the consequence of a shift in analysis workflow, fromextract, transform, load (ETL) to extract, load, transform (ELT). A Note on the Shift from ETL to ELT. In the past, data movement was defined by ETL: extract, transform, and load. Extract, load, Transform (ELT) tools.

Data Warehouse 52

Data Warehouse ETL Tableau Cloud Data 52

phData

OCTOBER 17, 2024

Apache Airflow Airflow is an open-source ETL software that is very useful when paired with Snowflake. Airflow is entirely in Python, so it’s relatively easy for those with some Python experience to get started using it. Airflow uses Directed Acyclic Graphs (DAGs) to represent workflows as tasks with defined dependencies.

AI 52

AI AI Data Quality SQL 52

ODSC - Open Data Science

DECEMBER 9, 2024

Additionally, a clear majority of current projects ( 85% to be exact) leverage open-source programming languages like Python and R rather than proprietary options. At the core are versatile open-source languages like Python and R that provide accessible foundations for statistical analysis and model building.

Data Science 40

Data Science Python Machine Learning Machine Learning 40

DagsHub

APRIL 7, 2024

This also means that it comes with a large community and comprehensive documentation. Thanks to its various operators, it is integrated with Python, Spark, Bash, SQL, and more. Flexibility: Its use cases are wider than just machine learning; for example, we can use it to set up ETL pipelines. It is lightweight.

Machine Learning 59

Machine Learning Machine Learning ML ML 59

The MLOps Blog

OCTOBER 20, 2023

References : Links to internal or external documentation with background information or specific information used within the analysis presented in the notebook. You could link this section to any other piece of documentation. documentation. If a reviewer wants more detail, they can always look at the Python module directly.

SQL 52

SQL Data Scientist Database Python 52

Pickl AI

JULY 3, 2023

Explore their features, functionalities, and best practices for creating reports, dashboards, and visualizations. Develop programming skills: Enhance your programming skills, particularly in languages commonly used in BI development such as SQL, Python, or R.

Business Intelligence 52

Business Intelligence Business Intelligence SQL Data Visualization 52

DagsHub

OCTOBER 23, 2024

For instance, if the collected data was a text document in the form of a PDF, the data preprocessing—or preparation stage —can extract tables from this document. The pipeline in this stage can convert the document into CSV files, and you can then analyze it using a tool like Pandas.

Machine Learning 52

Machine Learning Machine Learning Data Lakes AI 52

databricks

JUNE 12, 2025

that evolution continues with major advances in streaming, Python, SQL, and semi-structured data. Data Engineer, 84.51° What’s Next Stay tuned for more details in the Apache Spark documentation. With the recent release of Apache Spark 4.0, You can read more about the release here.

SQL 257

SQL Data Engineer Data Engineering Data Engineering 257

IBM Journey to AI blog

JULY 17, 2023

These encoder-only architecture models are fast and effective for many enterprise NLP tasks, such as classifying customer feedback and extracting information from large documents. With multiple families in plan, the first release is the Slate family of models, which represent an encoder-only architecture.

AI 74

AI AI Machine Learning Machine Learning 74

The MLOps Blog

DECEMBER 7, 2022

And that’s when what usually happens, happened: We came for the ML models, we stayed for the ETLs. But even when the ETLs were well thought out, they were a bit “outdated” in their approach. ETL Pipeline ETL Pipeline | Source: Author The pipeline is triggered by Eventbridge , and can be done either manually or by cron.

ML 52

ML ML ETL AWS 52

ODSC - Open Data Science

SEPTEMBER 29, 2023

Spark is more focused on data science, ingestion, and ETL, while HPCC Systems focuses on ETL and data delivery and governance. It’s not a widely known programming language like Java, Python, or SQL. ECL sounds compelling, but it is a new programming language and has fewer users than languages like Python or SQL.

Data Lakes 52

Data Lakes Clustering Big Data Big Data 52

phData

JUNE 26, 2023

Advanced Data Processing Capabilities KNIME provides a wide range of nodes for data extraction, transformation, and loading (ETL), but it also offers advanced data manipulation and processing capabilities. This includes machine learning , statistical modeling, and text mining, among others.

Tableau 52

Tableau Data Preparation Machine Learning Machine Learning 52

The MLOps Blog

AUGUST 3, 2023

At a high level, we are trying to make machine learning initiatives more human capital efficient by enabling teams to more easily get to production and maintain their model pipelines, ETLs, or workflows. You could almost think of Hamilton as DBT for Python functions. It gives a very opinionary way of writing Python.

ML 52

ML ML Data Scientist Machine Learning 52

phData

SEPTEMBER 4, 2024

In that case, ThoughtSpot also leverages ELT/ETL tools and Mode, a code-first AI-powered data solution that gives data teams everything they need to go from raw data to the modern BI stack. However, ThoughtSpot can often work around these issues using custom connectors, ETL/ELT processes, or APIs, bridging the gap between the two.

Analytics 52

Analytics Analytics ETL SQL 52

phData

FEBRUARY 14, 2023

Tips When Considering Streamsets Data Collector: As a Snowflake partner, Streamsets includes very intricate documentation on using Data Collector with Snowflake, including this book you can read here. This allows users to utilize Python to customize transformations. Data Collector can use Snowflake’s native Snowpipe in its pipelines.

Data Warehouse 59

Data Warehouse Azure AWS Database 59

phData

OCTOBER 11, 2023

See the Power BI documentation. Data Processing Within KNIME’s toolkit, you’ll find an extensive array of nodes catering to data extraction, transformation, and loading (ETL). You can, however, code in Python, R, Java, JavaScript, or CSS within KNIME if you want. Execute the workflow. Check your Power BI Workspace!

Power BI 45

Power BI Data Preparation Analytics Data Warehouse 45