Data Pipeline

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. It uses ...

Apache NiFi is an open-source data integration tool designed to automate the flow of data between systems. It provides a...

Apache Oozie is a workflow scheduler system designed to manage Hadoop jobs. It allows users to define complex workflows ...

Argo Workflows is a Kubernetes-native workflow engine designed to orchestrate parallel, containerized jobs. It enables u...

Couler provides a unified Python-based interface for defining and managing machine learning workflows across multiple wo...

Dagster is an open-source data orchestrator designed for building, testing, and running data pipelines. It focuses on de...

DBT (Data Build Tool) is an open-source tool that enables data analysts and engineers to transform data in their warehou...

Flyte is a cloud-native workflow automation platform designed for machine learning and data processing. It enables users...

Genie is a job orchestration engine designed to manage and trigger the execution of jobs in Hadoop-based systems. It pro...

Hamilton is a Python-based micro-orchestration framework designed to define and manage dataflows, such as feature engine...

Instill VDP is an open-source, self-hostable data pipeline tool designed to automate and manage end-to-end data processi...

Instructor is an open-source tool that simplifies extracting structured data (e.g., JSON) from large language models (LL...

Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data pipelines. It provide...

Luigi is a Python-based workflow management system designed to automate and orchestrate complex data pipelines. It handl...

Metaflow is an open-source framework designed to streamline the development and orchestration of data science workflows,...

Pachyderm is an open-source data pipeline platform built on Kubernetes, designed to automate and manage machine learning...

Pixeltable is an open-source Python library designed to simplify the creation and management of multimodal AI data pipel...

Ploomber is an open-source framework for building and managing data pipelines, enabling iterative development and deploy...

Prefect Core is an open-source workflow management system designed for orchestrating and automating data pipelines. It p...

SeqIO is a library for processing sequential data to be fed into downstream sequence models, providing a standardized in...

Snakemake is an AI agent in the Data Pipeline category. ![](https://img.shields.io/github/stars/snakemake/snakemake.svg?...

Towhee is an AI agent in the Data Pipeline category. ![](https://img.shields.io/github/stars/towhee-io/towhee.svg?cacheS...

unstructured is an AI agent in the Data Pipeline category. ![](https://img.shields.io/github/stars/Unstructured-IO/unstr...

ZenML is an AI agent in the Data Pipeline category. ![](https://img.shields.io/github/stars/zenml-io/zenml.svg?cacheSeco...