Apache Airflow 1.8: What's New, Release History & Support Lifecycle

What Is New in Apache Airflow 1.8

Apache Airflow 1.8 brings meaningful changes to the scheduler, pool management, DAG lifecycle defaults, and Google Cloud integrations. Several configuration keys have been renamed and new behavioral defaults introduced, making this a release that demands careful review before upgrading in production.

What Database and System Changes Are Required Before Upgrading to Airflow 1.8?

Airflow 1.8 requires a database schema migration and updated systemd unit files before the new version can run correctly.

Before starting any Airflow 1.8 instance, shut down all Airflow processes, take a full database backup, and run:

airflow upgradedb

If you manage Airflow with systemd, the unit files have also changed in this release. Deploy the updated unit files from the Airflow distribution before restarting services. Note that the webserver process does not detach cleanly in 1.8 -- this is a known issue flagged for a future fix.

In practice, teams that skip the schema upgrade will encounter startup failures. The database migration is non-negotiable and should be the very first step in your upgrade runbook.

How Did the Airflow 1.8 Scheduler Architecture Change and What New Options Should You Configure?

The 1.8 scheduler now processes each DAG in its own subprocess, improving fault isolation and making scheduler log management a new operational concern.

Each DAG file gets its own scheduler log written to child_process_log_directory, which defaults to <AIRFLOW_HOME>/scheduler/latest. You will need a log rotation policy for this directory -- these files are not cleaned up automatically and will grow over time.

Several scheduler configuration keys changed behavior or were introduced fresh:

• run_duration -- replaces the old loop-count model. Defaults to -1 (run continuously). The scheduler now terminates after this many seconds rather than after a fixed number of loops.
• num_runs -- now means the number of scheduling attempts per DAG file within run_duration, not total scheduler loops. Defaults to -1 (try indefinitely).
• min_file_process_interval -- controls how frequently updated DAG files are re-read from disk.
• dag_dir_list_interval -- controls how often the scheduler scans the DAG folder for new files. Decrease this during development if newly added DAGs are not appearing promptly.
• catchup_by_default -- new global flag. When set to False, the scheduler only runs the most recent interval for a DAG rather than backfilling all missed runs. Can also be set per-DAG with catchup = False.

Watch out for the known issue with num_runs = -1: some users have reported spurious task-parsing errors under this default. If you hit this, change the default to None in cli.py as a temporary workaround while the fix is tracked upstream.

How Does Airflow 1.8 Change Default DAG Behavior at Creation and Scheduling?

Two behavioral defaults shift significantly in 1.8: new DAGs are now paused on creation, and the scheduler is stricter about dynamic start_date values.

Previously, a freshly imported DAG would begin scheduling immediately. In 1.8, all new DAGs are paused at creation. This is the safer default for production environments where you want explicit control over when a DAG starts running. To restore the old behavior, add the following to airflow.cfg:

[core]
dags_are_paused_at_creation = False

The scheduler is also less tolerant of dynamic start_date values such as start_date = datetime.now(). This pattern was never recommended, but 1.7 was forgiving about it. In 1.8, DAGs using dynamic start dates may silently fail to schedule. The fix is to switch to a fixed start_date and rename the DAG to clear its historical schedule state -- renaming is required because the old schedule metadata will otherwise interfere.

This matters if you have any DAGs -- often quick prototypes or utility DAGs -- where someone took a shortcut with datetime.now(). Audit your DAG folder before upgrading.

What Happened to Pool Enforcement and Why Were Tasks Blocked After the Upgrade?

Airflow 1.7.1 had a bug that allowed more pool slots to be consumed than the pool actually contained; 1.8.0 enforces pool limits correctly, which can cause tasks to stall after upgrading.

If you were unknowingly over-subscribing a pool -- which 1.7.1 silently permitted -- those queued tasks will now correctly wait for a free slot. This can look like tasks are stuck even though all their upstream dependencies are satisfied. There are two workarounds:

• Temporarily increase the pool's slot count above the number of queued tasks until the backlog clears.
• Move the affected tasks to a new, larger pool.

In practice, most teams will not hit this unless they deliberately undersized pools and relied on the lenient behavior. Check your pool utilization metrics before the upgrade to avoid a surprise freeze on day one.

What Breaking Changes Affect Google Cloud Operators and Hook Authentication in Airflow 1.8?

All Google Cloud Operators and Hooks are unified in 1.8 to use a single client library and a single connection type.

If you have existing Google Cloud connections in the Airflow metadata database, verify that each connection has its type set to Google Cloud Platform. Connections configured with the old type may fail to authenticate after the upgrade.

More critically, P12 key file authentication is no longer supported. Only JSON service account key files are accepted. If your connections use P12 keys, you must generate new JSON keys in your Google Cloud project and update every affected connection before upgrading:

# Old: P12 key file (no longer supported in 1.8)
# New: JSON key file required
# In Airflow UI: Admin -> Connections -> Edit each GCP connection
# Set "Keyfile JSON" field with the contents of your .json service account key

This matters if you have automated pipelines loading BigQuery, writing to GCS, or triggering Dataflow jobs -- any of those will break at runtime if the key format is not migrated beforehand.

Which Operator Import Patterns Were Deprecated and Will Break in Airflow 2.0?

Airflow 1.8 removes support for importing operators from the top-level airflow.operators namespace and stops accepting arbitrary constructor arguments in operators.

The top-level shortcut imports that many DAGs relied on are now gone. Update all DAG files to use explicit submodule imports:

# Broken in 1.8 -- do not use
from airflow.operators import PigOperator

# Correct -- import from the specific submodule
from airflow.operators.pig_operator import PigOperator

Additionally, Operator.__init__() previously swallowed any extra positional or keyword arguments without error. In 1.8, passing unrecognized arguments to an operator constructor raises an exception. This catches typos and misconfigured operators that were silently ignored before.

Most teams will find these import issues quickly via a syntax check or DAG parse run. Running python your_dag.py for each file is the fastest way to surface broken imports before you push to production.