Airflow Architecture (Scheduler, Webserver, Executor)

What is Airflow Architecture?

Airflow’s architecture is the backbone that supports its ability to manage workflows—think of it as the engine room of a ship, where every part has a job to keep things moving. It’s built around a few key players: the Scheduler, which decides when tasks run; the Webserver, which lets you see what’s happening; and the Executor, which does the actual work. These components rely on a metadata database to keep track of everything and work together to turn your Python scripts—called Directed Acyclic Graphs (DAGs), detailed in Introduction to DAGs in Airflow—into actionable, scheduled tasks. Whether you’re running a simple script with BashOperator or integrating with Airflow with Apache Spark, this architecture ensures everything runs smoothly. Let’s break it down piece by piece.

The Scheduler

The Scheduler is the mastermind of Airflow’s timing, ensuring your tasks kick off exactly when they’re supposed to. It’s a background process that’s always running, constantly checking your workflows to see what needs to happen next.

How the Scheduler Works

Imagine the Scheduler as a tireless clock-watcher. You start it with a command—airflow scheduler, as shown in Airflow CLI: Overview and Usage—and it springs into action. Its job is to scan the dags folder (e.g., ~/airflow/dags, set up via DAG File Structure Best Practices) where your Python DAG scripts live. Each DAG has a schedule_interval—like @daily or 0 0 * * *—telling it how often to run, detailed in DAG Scheduling (Cron, Timetables). The Scheduler reads these scripts, figures out when each task is due based on that interval and the start_date, and queues them up for execution. It’s not just about timing—it also checks dependencies, so if Task A needs to finish before Task B, it won’t start B too soon, as explained in Task Dependencies (set_upstream, set_downstream).

For example, say you have a DAG:

Copy codefrom airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG(
    dag_id="daily_check",
    start_date=datetime(2025, 1, 1),
    schedule_interval="@daily",
) as dag:
    start = DummyOperator(task_id="start")
    end = DummyOperator(task_id="end")
    start >> end

The Scheduler sees @daily, calculates that it runs at midnight each day starting January 1, 2025, and queues start followed by end when the clock hits that time. It’s always on, keeping your workflows on track—learn more in Introduction to Airflow Scheduling.

Why the Scheduler Matters

Without the Scheduler, your DAGs would just sit there—beautiful Python scripts with no life. It’s what breathes automation into Airflow, making sure tasks don’t need manual triggers. It’s smart too—if a task fails, it can retry it based on settings like retries (see Task Retries and Retry Delays), and it handles backfills for past runs. This constant vigilance keeps your pipelines humming, whether they’re daily ETL jobs or weekly ML updates.

Configuring the Scheduler

You don’t need much to get the Scheduler going—just run airflow scheduler after setting up Airflow, as outlined in Installing Airflow (Local, Docker, Cloud). It uses settings from airflow.cfg—like how often it scans DAGs (default is every 30 seconds)—and you can tweak these in Airflow Configuration Options. For bigger setups, optimize it with Reducing Scheduler Latency to keep it snappy.

The Webserver

The Webserver is your window into Airflow’s world, serving up a web-based interface where you can watch your workflows unfold. It’s the visual hub that makes Airflow user-friendly.

How the Webserver Works

You launch the Webserver with airflow webserver -p 8080—another handy command from Airflow CLI: Overview and Usage—and it starts a server on your machine, typically at localhost:8080. Open your browser, type that address, and you’re greeted with Airflow’s UI. This interface pulls data from the metadata database (set up in Airflow Metadata Database Setup) to show you every DAG in your dags folder. You’ll see their status—running, succeeded, failed—along with a graph view of tasks and their dependencies, detailed in Airflow Graph View Explained. Click a task, and you can peek at its logs, stored in ~/airflow/logs (see Task Logging and Monitoring), or retry it if it flopped.

For that daily_check DAG, the Webserver lists it under “DAGs,” shows green circles when tasks succeed, and lets you dig into why end might’ve failed. It’s your real-time dashboard, pulling everything together.

Why the Webserver Matters

The Webserver turns Airflow from a command-line tool into something you can actually see and interact with. Without it, you’d be stuck tailing logs or guessing what’s happening—fine for robots, not humans. It’s not just pretty; it’s practical—you can trigger DAGs manually with Triggering DAGs via UI, pause them with Pause and Resume DAGs, or check statuses in Monitoring Task Status in UI. It’s your control center, making Airflow accessible.

Configuring the Webserver

Starting it is simple—just run airflow webserver -p 8080 after installation. The -p 8080 sets the port—change it to 8081 if 8080’s busy (e.g., airflow webserver -p 8081). It reads from airflow.cfg, where you can adjust settings like authentication—customize it with Customizing Airflow Web UI or secure it via Security Best Practices in Airflow. Keep it running alongside the Scheduler for full functionality.

The Executor

The Executor is the muscle of Airflow, taking the tasks the Scheduler queues and actually running them. It’s what turns plans into action.

How the Executor Works

When the Scheduler says, “It’s time,” the Executor steps up. Airflow offers different Executors—SequentialExecutor, LocalExecutor, CeleryExecutor, and more, detailed in Airflow Executors (Sequential, Local, Celery). The SequentialExecutor, default for local setups, runs one task at a time on your machine—simple but slow for big jobs. LocalExecutor uses your machine’s CPU cores to run multiple tasks at once, while CeleryExecutor spreads them across worker machines using a queue like RabbitMQ, perfect for scale. You set this in airflow.cfg under [core] executor.

Take our daily_check DAG: the Executor runs start, waits for it to finish (SequentialExecutor), then runs end. With LocalExecutor, it might run both if resources allow—CeleryExecutor could send end to another machine. It’s the doer, executing Python functions with PythonOperator or shell commands with BashOperator.

Why the Executor Matters

The Executor is what gets stuff done—without it, the Scheduler’s just a planner with no hands. Its type dictates how fast and how many tasks Airflow can handle. SequentialExecutor is fine for testing, but LocalExecutor or CeleryExecutor (or even Airflow with Kubernetes Executor) powers real workloads. It’s the difference between a solo worker and a team—crucial for performance, as explored in Airflow Performance Tuning.

Configuring the Executor

Out of the box, Airflow uses SequentialExecutor—great for Installing Airflow (Local, Docker, Cloud) beginners. To switch, open airflow.cfg (in ~/airflow), find [core], and change executor = SequentialExecutor to executor = LocalExecutor or executor = CeleryExecutor. For Celery, set up a broker (e.g., RabbitMQ) and workers—see Airflow with Celery Executor. Restart the Scheduler and Webserver after changes.

The Metadata Database

The metadata database is Airflow’s memory, storing all the details about your workflows so nothing gets lost.

How the Metadata Database Works

Set up with airflow db init (from Airflow Metadata Database Setup), this database—SQLite by default, or PostgreSQL for scale—tracks task states (running, failed, success), run times, and DAG history. The Scheduler writes when a task starts, the Executor updates its status, and the Webserver reads it to show you what’s up. It’s the glue holding everything together, persisting data in ~/airflow/airflow.db or a server for PostgreSQL.

Why the Metadata Database Matters

Without this database, Airflow would forget everything—task failures, past runs, all gone. It’s what lets you retry tasks, review logs, or backfill old runs with Catchup and Backfill Scheduling. For big setups, a robust backend like PostgreSQL (tuned with Database Performance in Airflow) keeps it reliable.

How It All Fits Together

Picture this: you write a DAG in ~/airflow/dags—say, our daily_check. The Scheduler spots it, sees @daily, and at midnight, queues start. The Executor (say, LocalExecutor) runs start, updates the metadata database to “success,” then runs end. The Webserver pulls this from the database, showing green circles at localhost:8080. If end fails, the database logs it, the Webserver displays red, and you retry it—all synced up. It’s a dance of timing, action, and visibility, orchestrated by these components.

FAQ: Common Questions About Airflow Architecture

Let’s tackle some questions folks often ask online—like on Stack Overflow or Reddit—about how Airflow’s architecture works, with detailed answers to clear things up.

First up, people wonder what happens if the Scheduler stops running. If you kill the airflow scheduler process—say, by closing its terminal—no new tasks get queued. Existing tasks finish if the Executor’s already running them, but nothing new starts until you restart it with airflow scheduler. The Webserver keeps showing past states from the database, but it won’t update with new runs—keep it alive as shown in Airflow CLI: Overview and Usage.

Another common one is whether the Webserver is required to run workflows. Nope—it’s optional for execution. The Scheduler and Executor can run DAGs without it, but you’d miss the UI. You’d be stuck checking logs manually in ~/airflow/logs (via Task Logging and Monitoring) instead of seeing everything in Airflow Web UI Overview. It’s like flying blind—possible, but why?

Folks also ask how the Executor knows which tasks to run. The Scheduler tells it—when it queues a task, it writes to the metadata database (e.g., “run start now”). The Executor—whether Sequential, Local, or Celery—reads that queue and acts, picking up tasks based on its type and capacity, as detailed in Airflow Executors (Sequential, Local, Celery). It’s a handoff, not a guess.

A big question is what’s the difference between the Scheduler and Executor. The Scheduler plans—it decides when tasks run based on schedule_interval and dependencies, queuing them up. The Executor does—it takes those queued tasks and runs their code, like a Python function or shell script. The Scheduler’s the brain, the Executor’s the hands—together, they make Airflow work, with roles split for efficiency.

Finally, people ask how to scale Airflow’s architecture. Start with the Executor—switch from Sequential to LocalExecutor for parallelism on one machine, or CeleryExecutor for multiple machines (see Airflow with Celery Executor). Use PostgreSQL instead of SQLite for the database to handle more load (Database Performance in Airflow), and optimize the Scheduler with Reducing Scheduler Latency. It’s about picking the right tools for your workload.

Conclusion

Airflow’s architecture—Scheduler, Webserver, Executor, and metadata database—is a symphony of parts that turn your DAGs into reality. The Scheduler times it, the Executor runs it, the Webserver shows it, and the database remembers it. Together, they make Airflow a leader in workflow orchestration. Ready to set it up? Start with Installing Airflow (Local, Docker, Cloud), write DAGs in Defining DAGs in Python, and monitor them with Monitoring Task Status in UI. Dive deeper with Airflow Concepts: DAGs, Tasks, and Workflows!