Apache Airflow notes

Apache Airflow in the Modern Orchestration Ecosystem

This guide goes beyond basics and positions Apache Airflow within the entire orchestration ecosystem including legacy enterprise schedulers and modern workflow engines.

1. Scheduler & Orchestrator Ecosystem

Tool	Type	Strength	Limitation
Cron	Basic Scheduler	Simple	No dependencies
AutoSys	Enterprise Scheduler	Job control + dependency trees	Rigid, not developer-friendly
Control-M	Enterprise Scheduler	Enterprise reliability	Expensive, GUI-driven
Airflow	Workflow Orchestrator	Code-first DAG system	Batch-focused
Dagster	Data Orchestrator	Data-aware pipelines	Newer ecosystem
Prefect	Modern Orchestration	Simplified developer UX	Less mature
Temporal	Workflow Engine	Stateful workflows	Different paradigm

👉 Airflow sits between enterprise schedulers and modern developer-first orchestrators.

2. Evolution of Orchestration

graph LR A[Cron] --> B[AutoSys / Control-M] B --> C[Airflow] C --> D[Dagster / Prefect] D --> E[Temporal]

3. Why Airflow is Still Critical

Airflow dominates because:

• Massive ecosystem
• Python-native
• Battle-tested in production
• Flexible DAG definition

Even with new tools, Airflow remains the industry standard for batch orchestration.

4. Architecture Diagram

graph TD A[DAG Code] --> B[Scheduler] B --> C[Executor] C --> D[Workers] B --> E[Metadata DB] D --> E E --> F[Web UI]

5. Airflow Operator Classification

Block Diagram

graph TD A[Operators] A --> B[Action Operators] A --> C[Transfer Operators] A --> D[Sensor Operators] A --> E[Branch Operators] A --> F[Custom Operators] B --> B1[PythonOperator] B --> B2[BashOperator] C --> C1[S3ToRedshift] C --> C2[MySQLToHive] D --> D1[FileSensor] D --> D2[HttpSensor] E --> E1[BranchPythonOperator] F --> F1[User Defined]

6. Operator Categories Explained

1. Action Operators

Perform actual execution logic.

2. Transfer Operators

Move data between systems.

3. Sensor Operators

Wait for external conditions.

Sensors can cause resource starvation if poorly designed.

4. Branch Operators

Enable conditional workflow execution.

5. Custom Operators

Extend Airflow to match business requirements.

7. Airflow vs Temporal (Critical Concept)

Aspect	Airflow	Temporal
Execution Type	Batch	Event-driven
State Handling	External DB	Built-in durable state
Use Case	Data pipelines	Microservices workflows

Temporal is NOT a replacement — it solves a different problem.

8. Advanced Edge Cases

• Scheduler lag due to heavy DAG parsing
• Zombie tasks not cleaned properly
• Sensor deadlocks
• Backfill overloads system
• DB bottleneck under high concurrency

9. DAG Example

task1 >> task2

Always keep DAG logic minimal — push logic into tasks.

10. Future of Orchestration

• Hybrid orchestration (Airflow + Temporal)
• Dagster adoption in data teams
• Kubernetes-native pipelines

FAQs

Is Airflow outdated? → No, still dominant

Should I learn Temporal? → Yes for backend workflows

Best alternative? → Depends on use case

Introduction - FullStack Developer/ Data Engineer

Thank you for the briefing. Now, I can understand more about the position. After listening to the conversation, I am more confident to say that, “I am best fit for this role.”

Introduction

I am a Full-Stack Python Developer, with all years of experience, with around 80% into Backend work and 20% into the frontend.

Backend Expertise

In Backend, I worked on:

• Creating standalone scripts for automation, scheduled jobs, ETL jobs, and data pipelines.
• Building RESTful APIs and web applications.

Frontend Expertise

In Frontend, I worked on:

• Creating dashboards, graphs, and charts using d3.js or Highcharts libraries.
• Building tables using ag-grid.
• Working with JavaScript, jQuery, and React.

Databases

In databases:

• Relational Databases: MySQL, MS SQL, Oracle DB, PostgreSQL.
• NoSQL Databases: MongoDB, Cassandra.
• Data Warehousing: Snowflake schema.

Python Expertise

In Python:

• Creating web applications and RESTful APIs using Django, Flask, and FastAPI frameworks.
• Process automation and integrating with infrastructure (Linux/Windows).
• Data gathering from:
  1. Structured datasources like REST APIs (internal or external), databases, etc.
  2. Unstructured datasources like web scraping using Beautiful Soup.
  3. Structured, semi-structured, or unstructured file types like CSV, Excel, JSON, YAML, Parquet, etc.
• Following TDD (Test-Driven Development) by creating unit tests and integration tests using unittest or pytest modules.

Caching and Scheduling

In caching:

• Worked with Redis and Memcache.

In scheduling:

• Worked with Celery.
• Integrated Celery with Django applications for periodic jobs.

For data job orchestration, I worked with Airflow.

Public Cloud Experience

In Public Cloud, I am mostly associated with AWS Cloud. In AWS Cloud, I worked with:

Server-Based Architectures

• EC2 Instances
• Elastic Beanstalk
• Elastic Load Balancer
• Auto-Scaling
• Route53

Storage Solutions

• S3 Bucket for file storage
• S3 Glacier for archival storage

Databases

• AWS RDS & Redshift for relational databases
• AWS DynamoDB for NoSQL

Serverless Architectures

• AWS Lambda (time-triggered or event-triggered)
• AWS API Gateway (HTTP-triggered)
• AWS Event Scheduler for scheduling Lambda

Container-based Environments

• Created docker.yaml files for creating containers
• AWS EKS (Elastic Kuberntes Service) for Orchestrating the pods.
• Also, wrote the helm chart

For OAuth, I worked with AWS Cognito.

Also worked with SQS, SNS, and SES.

For big data processing, I worked with EMR clusters for PySpark.

For ETLs, I worked with AWS Glue Jobs with DataCatalog and PySpark.

CI/CD and Infrastructure as Code

Experience in creating CI/CD setups using Jenkins and Groovy scripts.

In terms of Infrastructure as Code, I worked mainly with:

• CloudFormation templates
• Terraform
• Pulumi Python module

Agile Methodologies

Experience in agile methodologies like Scrum and Kanban in facilitating agile ceremonies like:

• Daily standups
• Sprint reviews
• Planning sessions
• Retrospectives

Familiar with Agile tools like Jira and skilled in working with cross-functional teams including Dev teams, QA testers, and PM/POs.