Apache Airflow 2: What's New, Release History & Support Lifecycle

Can Airflow 2 run tasks on multiple executors at the same time?

Yes -- starting with Airflow 2.10.0, Airflow can run tasks across multiple executors concurrently, a capability described in AIP-61 (Hybrid Execution). Individual tasks or entire DAGs can be pinned to a specific executor, so a single DAG can mix a KubernetesExecutor task with a CeleryExecutor task without any workarounds.

In practice this means teams running mixed workloads -- for example, CPU-bound ML training alongside lightweight Python scripts -- no longer need separate Airflow deployments or wrapper hacks. The executor is exposed as a first-class field on operators and in the database.

# Pin a task to a specific executor
@task(executor="KubernetesExecutor")
def train_model():
    ...

@task(executor="LocalExecutor")
def notify_slack():
    ...

Watch out for the "experimental" label on this feature in 2.10.0 -- while it works in production, some edge cases around metrics and executor-specific configs were still being ironed out in patch releases. The executor field is stored in the task_instance table, so a fresh airflow db migrate is required after upgrading.

How did Airflow 2 change the way dataset-based scheduling works?

Airflow 2.9 introduced a significant behavioral fix: datasets no longer trigger paused or inactive DAGs. Previously, events that accumulated while a DAG was paused would fire the DAG the moment it was unpaused, creating surprise runs that confused on-call engineers. The new behavior means a dataset schedule is only satisfiable by events that arrive while the consuming DAG is active.

The 2.9 release also introduced conditional (logical) dataset expressions, letting you combine datasets with AND / OR semantics to build more nuanced trigger conditions. The 2.10 release added Dataset Aliases, which allow dynamic emission -- a task can emit a dataset event to an alias and have it resolve to one or more actual datasets at runtime, decoupling producer and consumer definitions.

from airflow.datasets import Dataset, DatasetAlias

# Conditional: trigger only when BOTH datasets are updated
with DAG(
    dag_id="consume_both",
    schedule=(Dataset("s3://bucket/a") & Dataset("s3://bucket/b")),
):
    ...

# Alias: producer emits to a logical name
@task(outlets=[DatasetAlias("reporting.daily")])
def produce():
    ...

This matters if your pipelines rely on dataset-triggered DAGs with a lot of pause/unpause cycles -- audit your run history after upgrading to ensure no unexpected backlog of events exists. Use the new REST API endpoint (POST /datasets/events) or the UI button to manually emit dataset events during testing.

What changed about task try_number and execution lifecycle in Airflow 2.10?

In Airflow 2.10.0, the try_number is now fixed at scheduling time and never incremented during execution. Before this change, try_number was bumped at the start of each task run on the worker, which created confusing behavior when tasks resumed from deferral or rescheduled sensors -- the try count would be inflated without representing a genuine retry.

The new model is simpler: the scheduler assigns try_number before the task enters the queue, it stays constant for the duration of that attempt, and only a genuine new retry increments it. As a side effect, users who call ti.run() or airflow tasks run directly will no longer see try_number increment -- Airflow now assumes all executions are scheduler-driven.

A related improvement in 2.10 ensures that teardown tasks are no longer skipped when a DAG run is manually set to "failed" or "success" -- as long as the corresponding setup task had already started. This is critical if you use setup/teardown pairs to manage ephemeral cloud infrastructure. The DAG will temporarily remain in a running state while teardown executes before reaching the terminal state.

with DAG("infra_dag", ...):
    setup_task = create_cluster.as_setup()
    teardown_task = delete_cluster.as_teardown()

    setup_task >> run_job >> teardown_task
    # teardown_task now runs even if DAG is manually failed
    # as long as setup_task completed

What observability improvements ship with Airflow 2.10?

Airflow 2.10.0 graduated OpenTelemetry (OTel) traces from experimental to stable. The feature, introduced in preview during earlier 2.x releases, now emits structured trace data for the scheduler loop, triggerer, executor, and DAG file processor -- in addition to per-DAG-run traces that span the entire execution lifecycle.

Most teams already use OTel for metrics (stabilized in 2.9.3); the addition of traces means you can correlate a slow DAG run directly to scheduler contention or executor queue depth in your APM tooling (Datadog, Honeycomb, Jaeger, etc.) without writing custom instrumentation. Key configuration items to check:

• Set [metrics] otel_on = True and configure otel_host / otel_port to enable trace export.
• Override the service name via OTEL_SERVICE_NAME -- hardcoded defaults were removed in 2.10.3.
• The OTEL_RESOURCE_ATTRIBUTES environment variable is now honored for enriching spans with deployment metadata.
• Metrics for CPU and memory usage per task were added in 2.10.0, emitted via the existing statsd / OTel pipeline.

AIP-62 (Hook Lineage Instrumentation) also landed in 2.10.0, adding a HookLineageCollector that captures dataset-level lineage from hooks automatically, feeding into the dataset dependency graph in the UI.

What breaking changes and deprecations should teams plan for when upgrading to Airflow 2.9 or 2.10?

Several changes in the 2.9/2.10 line require deliberate migration work before upgrading. The most impactful are listed below grouped by effort level.

High effort -- requires code or infrastructure changes:

• MSSQL removed (2.9.0): If your Airflow metadata database runs on SQL Server, you must migrate to PostgreSQL or MySQL before upgrading. A community migration script is available but carries no official support or warranty.
• Dataset URI validation (2.9.0): Dataset identifiers that do not conform to AIP-60 URI rules will be rejected or silently normalized. Identifiers using the URI auth section, case-sensitive scheme names, or non-standard formats must be updated in DAG files.
• Rendered Template Field length cap (2.9.0): Template fields exceeding 4096 characters are truncated. Use [core] max_template_field_length to raise the cap if your DAGs push large strings through template fields.

Low effort -- behavioral changes to be aware of:

• AirflowTimeoutError now inherits from BaseException -- except Exception blocks will no longer catch task timeouts. Switch cleanup logic to finally blocks.
• The /logout endpoint in FAB Auth Manager changed from GET to POST with CSRF protection. Any automation or health checks hitting this endpoint via GET will break.
• Inactive/paused DAGs will no longer accumulate dataset events, changing run behavior after a DAG is re-enabled.
• TaskFlow functions must not define context parameter defaults other than None. The scheduler now validates this at parse time and marks violating DAGs as broken.

# Before 2.9: could set non-None default on context var -- now invalid
@task
def my_task(execution_date="2024-01-01"):   # BROKEN in 2.9+
    ...

# Correct: always use None as default for context-injected params
@task
def my_task(execution_date=None):
    ...

Frequently Asked Questions about Apache Airflow 2

Does upgrading to Airflow 2.9 require changes to existing DAGs that use datasets?
Most DAGs will continue to work, but dataset identifiers that look like URIs must now conform to AIP-60 validation rules. Run a DAG parse in a staging environment first -- any non-conforming dataset URIs will surface as warnings or errors before they can affect production.

Is MSSQL still supported as an Airflow metadata database in Airflow 2?
No. Support for Microsoft SQL Server as the Airflow metadata database was removed in Airflow 2.9.0 following a community vote. Existing users must migrate to PostgreSQL or MySQL prior to upgrading using the community migration script available in the airflow-mssql-migration GitHub repository.

What is the fastest way to enable OpenTelemetry traces in Airflow 2.10?
Set otel_on to True under the metrics section in airflow.cfg, configure otel_host and otel_port to point to your OTLP collector, and restart all Airflow components. Traces for the scheduler loop and triggerer will begin emitting immediately; per-DAG-run traces appear once new DAG runs are triggered after the upgrade.

Can individual tasks in a single DAG be routed to different executors in Airflow 2.10?
Yes, this is exactly the use case for the new hybrid/multiple-executor feature introduced in 2.10.0. You pass the executor name as a string to the executor parameter on any operator or TaskFlow decorator, and the scheduler routes each task to the appropriate executor. The feature is still marked experimental but is usable in production with proper testing.

Does the try_number change in Airflow 2.10 affect existing retry logic or alert thresholds?
It can. Before 2.10, try_number was sometimes incremented for rescheduled sensors and deferred tasks even when no real retry occurred, inflating observed retry counts. After 2.10 the number is more accurate, so monitoring rules or custom code that compared try_number values may observe lower numbers for the same task behavior and need recalibration.

How do I use the new skip_if and run_if decorators introduced in Airflow 2.10?
Decorate a TaskFlow task with @task.run_if(lambda context: ...) or @task.skip_if(lambda context: ...) passing a callable that receives the task context and returns a boolean. These decorators replace common patterns of checking context variables inside task bodies and branching to a skip exception, making conditional task execution declarative. Note that these decorators are stripped before virtualenv tasks execute to avoid import issues in isolated environments.