AI Agents · Airflow

Apache Airflow 3 New Features

Apache Airflow 3 New Features is the work that turns raw data into decisions. The pipeline from "we have data" to "we have a model that runs in production" is the same in every.

John Kihiu12 min read

Airflow 3, released in 2025, is the first major rework of the scheduler and execution model since the project moved off the monolithic webserver-scheduler-worker split it had carried since Airflow 1. The headline change is the DAG Processor and the new Task Execution API: workers no longer need direct database access or the full Airflow package installed, which is the change most teams running Airflow on Kubernetes have been waiting for.

The Task Execution API and remote workers

Before Airflow 3, every worker process imported the Airflow core, connected directly to the metadata database, and needed the same Python environment as the scheduler. That coupling made polyglot pipelines and tightly locked-down production clusters painful. Airflow 3 introduces a Task Execution API — a stable HTTP interface between the task runner and the scheduler — so a task can now run in a container with none of Airflow's dependencies installed, communicating over the API instead of a direct DB connection. This is also what makes the new Edge Executor practical: workers can live outside the primary network and still pull work.

Why this matters for ERP-adjacent pipelines

If you run Airflow to pull nightly extracts from an ERP and land them in a warehouse, the worker no longer needs the same Python/Airflow version as the scheduler. You can run a thin, purpose-built extraction image and let the API mediate state, which shrinks the blast radius of a dependency upgrade.

DAG versioning finally exists

Airflow 2 always ran a DAG's current file against historical task instances, so if you changed a DAG's structure, the Grid view for old runs would misrepresent what actually executed. Airflow 3 adds DAG versioning: each parse of a DAG file is stored as a version, and historical runs render against the version that was active when they ran. This is a real operational fix, not a UI polish item — postmortems on failed backfills used to require checking git history to know what the DAG looked like on a given date.

Event-driven scheduling and asset triggers

Airflow 2.4 introduced Datasets; Airflow 3 renames the concept to Assets and expands it into a real event-driven scheduling model. DAGs can now trigger on asset updates coming from outside Airflow entirely — an external system can POST to the asset events API and wake a downstream DAG — rather than every trigger needing to originate from another Airflow task. Combined with the new AssetWatcher pattern for polling external systems, this closes a long-standing gap where Airflow could only really react to its own runs.

PYTHON · ASSET-TRIGGERED DAG
from airflow.sdk import Asset, dag, task

raw_orders = Asset("s3://landing/orders/")

@dag(schedule=[raw_orders], catchup=False)
def orders_curate():

    @task
    def transform():
        # runs whenever raw_orders is marked updated
        # by any producer, Airflow task or external system
        ...

    transform()

orders_curate()

The new DAG authoring SDK

Airflow 3 ships a slimmer airflow.sdk package aimed specifically at DAG authors, separate from the provider and execution internals. The practical effect is fewer accidental imports of scheduler-only code in DAG files, and a clearer contract for what's safe to use in a DAG versus what belongs in a provider or plugin. If your DAGs still import from airflow.models directly, migrating to the SDK surface is the low-effort part of a Airflow 3 upgrade.

Provider compatibility before you upgrade

Not every community provider had a 3.0-compatible release on day one. Before upgrading a production instance, check the provider changelog for anything you depend on — particularly older AWS, Snowflake, or JDBC-based providers — rather than upgrading core and finding out a DAG import fails at parse time.

A rebuilt UI on React and FastAPI

The webserver's Flask/Flask-AppBuilder UI is replaced with a FastAPI backend and a React frontend. Functionally this mostly matters for plugin authors — custom views and menu links built against the old Flask-AppBuilder API need to be ported to the new plugin interface. For most teams, the visible change is a faster, less cluttered Grid view and a asset-centric page that didn't really exist before.

Airflow 2Airflow 3
Workers need full Airflow + DB accessWorkers talk to Task Execution API, no DB access needed
Datasets, Airflow-only triggersAssets, external event triggers via API
Grid view renders current DAG file for old runsDAG versioning renders the version that actually ran
Flask-AppBuilder UIFastAPI + React UI

Wrapping up

The theme across Airflow 3 is decoupling: workers from the scheduler's Python environment, scheduling from Airflow-only events, and the Grid view from the fiction that a DAG file never changes. None of it requires rewriting existing DAGs to get value — the DAG versioning and Assets improvements apply to upgraded instances automatically — but plan the provider compatibility check before you flip a production scheduler to 3.x.

John Kihiu
Acumatica ERP Developer · Laravel Engineer

Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.