Flyte
Kubernetes-native orchestration for data and ML workflows
Flyte is profiled here as a Pipeline Orchestration tool for engineering teams. Read about features, pricing, and how it compares to related options in the tools directory.
Description
Flyte is an open-source orchestration platform, built at Lyft and now maintained by Union.ai under the Linux Foundation, that runs strongly typed data and machine learning workflows on Kubernetes. Workflows are written in Python with typed inputs and outputs, and Flyte versions each one, caches results, and reproduces runs so experiments stay traceable. It provisions Kubernetes resources per task for isolation and scale, which makes it a fit for teams running heavy data processing and model training pipelines.
Key Capabilities:
Strongly typed Python workflows with versioning and reproducibility
Kubernetes-native execution that isolates and scales each task
Result caching that skips recomputation across runs
Resource controls for CPU, memory, and GPU per task
Data lineage that tracks inputs and outputs across a pipeline
Multi-tenant projects with role-based access for shared clusters
Alternative tools
- Kestra
Declarative, event-driven orchestration defined in YAML
- Temporal
Durable execution for long-running, reliable workflows
- Prefect
Python-native orchestration for data and ML workflows
- Dagster
Asset-centric orchestration for data and ML pipelines
- Apache Airflow
The most widely deployed open-source workflow orchestrator