Airflow DAGs Scheduling Sensors and Backfills

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: make time-based data work explicit

Airflow is an orchestrator for data engineering work that must happen in a known order, at known times, and with a clear record of what ran. A DAG, or directed acyclic graph, describes tasks and dependencies: extract after a file appears, transform after extraction succeeds, publish after validation passes. Scheduling decides which logical data intervals should exist. Sensors wait for outside conditions. Backfills create or repair historical runs when code, source data, or scheduling policy changes.

The outcome of this lesson is practical: you should be able to read an Airflow DAG and predict when it creates runs, what each run represents, why a sensor is waiting, and how to backfill a date range without accidentally duplicating data or overwhelming an executor.

How Airflow thinks internally

Airflow stores DAG definitions as Python files, but it does not treat them as normal scripts that run top to bottom for each schedule. The scheduler repeatedly parses DAG files, builds an in-memory graph of task objects, and records metadata in the Airflow database. The scheduler then creates DagRun records for logical dates that match the timetable. For modern Airflow usage, the important idea is the data interval: a daily run dated 2026-03-10 usually processes the interval that started at 2026-03-10 00:00 and ended at 2026-03-11 00:00, not the instant named by the timestamp.

Each DagRun contains TaskInstance records, one per task for that logical date. A task instance moves through states such as scheduled, queued, running, success, failed, skipped, upstream_failed, or deferred. The scheduler only queues a task instance when its dependencies are satisfied: upstream tasks are complete, pools have capacity, retry delay has passed, the DAG is not paused, and concurrency limits allow more work.

A sensor is just a task with waiting behavior. In poke mode it occupies a worker slot while checking repeatedly. In reschedule mode it frees the worker slot between checks. Deferrable sensors go further: they hand the wait to Airflow’s triggerer process and resume when an asynchronous trigger fires. The choice matters because a fleet of file or partition sensors can consume the same worker capacity needed for real transformation tasks.

Backfilling is not a special kind of business logic. It is the act of asking Airflow to create and run DagRuns for older logical dates. A safe DAG therefore has to be parameterized by the logical date or data interval instead of by the wall-clock time at which the Python process happens to run.

DAG anatomy and scheduling syntax

A production DAG definition has several moving pieces. The DAG id names the workflow. The schedule can be a cron expression, a preset such as daily, a timedelta, a dataset trigger, or no schedule for manual runs. The start date is the first logical date Airflow may consider. The catchup flag controls whether missed scheduled intervals are created automatically. Default arguments can set owner, retries, retry delay, and notifications. Operators define units of work, and dependency operators such as >> attach edges between tasks.

from __future__ import annotations

import pendulum
from airflow import DAG
from airflow.operators.empty import EmptyOperator

with DAG(
    dag_id="warehouse_daily_minimal",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="0 3 * * *",
    catchup=False,
    tags=["warehouse"],
) as dag:
    extract = EmptyOperator(task_id="extract")
    transform = EmptyOperator(task_id="transform")
    publish = EmptyOperator(task_id="publish")

    extract >> transform >> publish

This DAG is eligible to run once per day at 03:00 UTC. With catchup=False, enabling it after many missed intervals creates only the latest scheduled run, not every historical day since 2026-01-01. The three tasks do no work, but their states demonstrate dependency behavior: transform cannot be queued until extract succeeds, and publish waits for transform.

Example 1: date-partitioned daily load

The first real design rule is to bind work to Airflow’s interval variables. A daily warehouse load should write the partition belonging to the run’s logical date, not whatever date the server clock returns. That makes retries and backfills deterministic.

from __future__ import annotations

import pendulum
from airflow.decorators import dag, task

@dag(
    dag_id="orders_daily_partition",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="@daily",
    catchup=True,
    max_active_runs=2,
)
def orders_daily_partition():
    @task
    def build_partition(data_interval_start=None, data_interval_end=None) -> str:
        partition = data_interval_start.format("YYYY-MM-DD")
        sql = (
            "insert overwrite table mart.orders partition (ds='{ds}') "
            "select * from raw.orders where created_at >= '{start}' "
            "and created_at < '{end}'"
        ).format(ds=partition, start=data_interval_start, end=data_interval_end)
        return sql

    build_partition()

orders_daily_partition()

For the run whose interval starts on 2026-01-05, the generated statement targets ds='2026-01-05' and filters records from the start of January 5 up to, but not including, January 6. If that run is retried two days later, it still processes January 5. max_active_runs=2 allows two daily intervals to run at once during catchup while preventing a large backlog from flooding the warehouse.

Example 2: waiting for an upstream file

Many pipelines cannot start until an upstream system lands a file. A sensor turns that external readiness check into an Airflow task state. The key is to wait on the date-specific path for the DagRun, not a generic latest directory.

from __future__ import annotations

import pendulum
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.sensors.filesystem import FileSensor

with DAG(
    dag_id="orders_file_sensor",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="@daily",
    catchup=True,
    max_active_runs=1,
) as dag:
    wait_for_orders = FileSensor(
        task_id="wait_for_orders",
        filepath="/data/landing/orders/{{ ds }}/orders.csv",
        poke_interval=60,
        timeout=60 * 60,
        mode="reschedule",
    )

    load_orders = BashOperator(
        task_id="load_orders",
        bash_command="python /opt/jobs/load_orders.py --date {{ ds }}",
    )

    wait_for_orders >> load_orders

For logical date 2026-01-07, the sensor checks /data/landing/orders/2026-01-07/orders.csv. If the file appears within an hour, the load task starts and receives --date 2026-01-07. If the file never arrives, the sensor times out and fails; downstream loading does not run. Reschedule mode is appropriate because a one-minute file check should not tie up a worker slot for the whole hour.

Example 3: controlled backfill

Suppose the transformation logic for orders changed and partitions from January 1 through January 7 must be rebuilt. The DAG above is backfillable because it uses {{ ds }} and interval-aware paths. The command should target a bounded range and the intended DAG.

airflow dags backfill orders_file_sensor \
  --start-date 2026-01-01 \
  --end-date 2026-01-07

The expected behavior is seven DagRuns, one for each daily logical date in that inclusive range, subject to the DAG’s active-run and task concurrency limits. Each run waits for its own dated file and loads its own dated partition. If January 3 is missing, the January 3 run fails or waits until timeout while the other dates can still complete according to scheduler capacity.

Design choices and trade-offs

Catchup versus no catchup. Catchup is useful for partitioned facts, slowly changing dimensions, and data sets where every interval matters. It is risky for APIs with rate limits or pipelines that write to non-partitioned destinations. For alerting jobs or latest-only snapshots, catchup=False is often the better expression of intent.

Cron time versus data interval. A cron expression says when a run should be scheduled; the data interval says what the run should process. Keep those ideas separate. A daily DAG scheduled at 03:00 often processes the previous calendar day because the interval has closed and upstream systems had time to deliver files.

Sensors versus event-driven triggering. Sensors are simple and visible in the Airflow graph, but many long waits can create metadata churn and operational noise. Dataset scheduling, object-storage notifications, or external event bridges can be cleaner when upstream systems can emit reliable events. Sensors remain useful when the upstream readiness condition must be polled.

Parallel backfills versus downstream pressure. Increasing max_active_runs, pool slots, or task concurrency can finish historical repair faster. It can also overload a warehouse, saturate source systems, or create lock contention on target tables. Backfills should be sized to the slowest shared dependency, not only to available Airflow workers.

Failure modes and troubleshooting

No runs are being created. Symptoms include a DAG visible in the UI with no recent DagRuns. Check whether the DAG is paused, whether the start date is in the future, whether the schedule is None, and whether the scheduler is parsing the file successfully. Correct the start date or schedule, unpause the DAG, and inspect import errors if parsing failed.

A backfill writes today’s partition repeatedly. The symptom is many historical DagRuns succeeding while only the current date changes in the warehouse. The usual cause is using datetime.now(), a database current_date, or an unparameterized latest path inside task logic. Diagnose by opening task logs for two different logical dates and comparing rendered SQL or command arguments. Correct the code to use {{ ds }}, data_interval_start, or explicit partition parameters.

Sensors block the worker queue. Symptoms include many running sensors and transformation tasks stuck queued. The cause is usually poke-mode sensors occupying workers during long waits. Confirm task states in the UI and inspect sensor configuration. Change suitable sensors to reschedule or deferrable mode, increase triggerer capacity for deferrable sensors, or replace polling with event-based scheduling.

Duplicate rows appear after retries. Airflow retries rerun task code; it does not automatically make writes idempotent. Diagnose by checking whether the target operation appends without a unique key, partition overwrite, merge condition, or run marker. Correct by writing each interval to a deterministic partition, using merge/upsert semantics, or staging output before an atomic publish step.

Security, reliability, and performance implications

Airflow templates can place dates, paths, and connection identifiers into commands. Avoid placing secrets directly in DAG files or rendered shell commands; use Airflow connections, variables with appropriate access controls, or a secret backend. Keep sensor paths specific enough that a run cannot accidentally consume another tenant’s or another date’s file.

Reliability comes from idempotent interval processing. A task should be safe to retry for the same logical date, and a backfill should rebuild the same output every time from the same inputs. Performance depends on scheduler load as well as execution load: thousands of tiny tasks or sensors can stress the metadata database even when worker CPU is idle. Use task grouping for readability, pools for shared systems, and batching when per-partition overhead dominates useful work.

Hands-on lab: build and backfill a local DAG

Prerequisites: a working Airflow environment, access to the Airflow CLI, and permission to create a DAG file in the configured DAGs folder. Use a non-production instance or a development namespace.

  1. Create a file named lab_orders_backfill.py in the DAGs folder.
  2. Add a daily DAG with start_date set to 2026-01-01, catchup=True, and one task that prints {{ ds }}.
  3. Run airflow dags list and confirm that the DAG id appears.
  4. Trigger a bounded backfill for 2026-01-01 through 2026-01-03.
  5. Open task logs or use the CLI to confirm that the three runs printed three different dates.
from __future__ import annotations

import pendulum
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator

with DAG(
    dag_id="lab_orders_backfill",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="@daily",
    catchup=True,
) as dag:
    print_date = BashOperator(
        task_id="print_date",
        bash_command="echo processing {{ ds }}",
    )
airflow dags list | grep lab_orders_backfill
airflow dags backfill lab_orders_backfill --start-date 2026-01-01 --end-date 2026-01-03

Verification: the logs should contain processing 2026-01-01, processing 2026-01-02, and processing 2026-01-03 in separate task instances. Cleanup: pause the lab DAG, delete the lab file from the DAGs folder, and remove lab DagRun records through the UI or CLI if your development environment keeps metadata between exercises.

Assessment exercises

  1. A daily DAG scheduled at 02:00 loads data using datetime.today(). Explain what breaks during a seven-day backfill and rewrite the parameter strategy.
  2. A file sensor waits six hours in poke mode while all warehouse tasks remain queued. Identify the resource being consumed and propose two corrections.
  3. You need to repair 180 daily partitions but the warehouse can handle only five concurrent loads. Which Airflow settings or constructs would you use to limit pressure?
  4. Given a DAG with start_date tomorrow and catchup=True, predict whether historical runs appear today and justify your answer.
  5. Design a sensor path for a customer-events pipeline so each logical date reads only its own input.

Summary

Airflow DAGs are executable dependency graphs backed by scheduler metadata. Scheduling creates logical data intervals, sensors model external readiness, and backfills replay older intervals through the same task graph. The practical discipline is to make every task depend on the run’s logical date, keep waits from consuming scarce execution slots, and limit historical repair to the capacity of the systems Airflow coordinates.