Course Outline
Overview of Apache Airflow
- The concept of workflow orchestration
- Principal capabilities and advantages of Apache Airflow
- Enhancements in Airflow 2.x and an overview of its ecosystem
Architectural Design and Fundamental Principles
- Scheduler, web server, and worker components
- Structure of DAGs, tasks, and operators
- Executor types and backends (Local, Celery, Kubernetes)
Deployment and Configuration
- Installing Airflow in local and cloud-based setups
- Tuning Airflow settings for different executors
- Initializing metadata databases and external connections
Interacting with the Airflow Interface and Command Line
- Reviewing the Airflow web dashboard
- Tracking DAG executions, individual tasks, and system logs
- Utilizing the Airflow CLI for administrative tasks
Creating and Administering DAGs
- Building DAGs utilizing the TaskFlow API
- Applying operators, sensors, and hooks
- Handling dependencies and defining scheduling frequencies
Connecting Airflow to Data and Cloud Platforms
- Linking with databases, APIs, and message brokers
- Executing ETL workflows via Airflow
- Cloud connectors: AWS, GCP, and Azure operators
Surveillance and Observability
- Analyzing task logs and performing real-time monitoring
- Integrating metrics with Prometheus and Grafana
- Setting up alerts and notifications via email or Slack
Hardening Apache Airflow
- Implementing Role-Based Access Control (RBAC)
- Authentication methods using LDAP, OAuth, and SSO
- Managing secrets with Vault and cloud-native secret stores
Scaling Apache Airflow
- Managing parallelism, concurrency, and task queuing
- Leveraging CeleryExecutor and KubernetesExecutor
- Deploying Airflow on Kubernetes using Helm
Production-Ready Best Practices
- Incorporating version control and CI/CD pipelines for DAGs
- Conducting tests and debugging DAG workflows
- Ensuring high availability and performance at scale
Problem Solving and Performance Tuning
- Investigating failed DAGs and specific tasks
- Enhancing DAG execution efficiency
- Identifying common errors and strategies for avoidance
Recap and Future Directions
Requirements
- Practical knowledge of Python coding
- Basic understanding of data engineering or DevOps principles
- Familiarity with ETL processes or workflow orchestration concepts
Target Audience
- Data scientists
- Data engineers
- DevOps and infrastructure specialists
- Software developers
Testimonials (7)
The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.
Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH
Course - Apache Airflow: Building and Managing Data Pipelines
The training was spot on. Very useful theory and exercices.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
Vladimir - PUBLIC COURSE
Course - Apache Airflow
The training was spot on in all aspects. Usefull theoretical aspects and exercises.
