Apache Airflow
data engineering
data platform
Prefect
kubernetes
data pipelines
pipeline orchestration
Can you feel the breeze? Data-aware scheduling and more new features in Airflow
Data Science Leuven — Belgium
23 May 2023
About this talk
Remember when you last used Airflow? You might have been scratching your head a few times. Recently Airflow has picked up a steady pace of releases packed with lots of interesting new features. In this talk we’ll go over them and see how they individually contribute to a new way to use Airflow.
- TaskFlow and recent improvements
- Dynamic Task Mapping
- Data-aware scheduling
- Deferrable operators
- Grid View & logging
- KubernetesPodOperator vs. KubernetesExecutor vs. KubernetesCeleryExecutor
- Official Helm chart
- Comparison with Prefect
Picture taken at the meetup
Stay in the loop
See you at the next one?
I announce upcoming talks on LinkedIn — that's also where most of the conference chatter happens. Slides and recordings land right here on the speaking page. If you'd rather follow along quietly, the RSS feed has every new post and talk.