Apache Airflow 1.7: What's New, Release History & Support Lifecycle

Summary

How does Airflow 1.7 protect sensitive data stored in connections and variables?

Airflow 1.7 ships Fernet-based encryption for Variables and the extra field of Connections, so passwords and API tokens are no longer stored in plain text in your metadata database. This is the single most operationally significant change in this release for teams that store credentials in Airflow's built-in secrets store.

To enable encryption, generate a Fernet key and add it to airflow.cfg:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# Paste the output into airflow.cfg:
[core]
fernet_key = YOUR_GENERATED_KEY_HERE

In practice, existing un-encrypted variables and connection extras will continue to work after upgrading -- Airflow reads them as plain text. New values written after the key is set will be encrypted automatically. Watch out for the transition period if you rotate the key: existing encrypted values must be re-encrypted before discarding the old key.

Alongside this, 1.7 fixes a long-standing embarrassment where the database password was printed to stdout during airflow initdb, resetdb, and upgradedb. That leak is now closed.

What new Google Cloud operators and hooks ship with Airflow 1.7?

Airflow 1.7 dramatically expands the Google Cloud operator surface, making it a much more complete solution for GCP-centric data pipelines. The most impactful additions are a set of BigQuery and Cloud Storage operators that cover the canonical ETL path from MySQL all the way to a BigQuery table.

New and improved GCP capabilities include:

• MySQL-to-GCS operator -- dumps MySQL query results to Cloud Storage; now handles TINYINT, INT24 (MEDIUMINT), and DATE types correctly.
• GCS-to-BigQuery operator -- loads files from GCS into BQ; defaults to 0 rows loaded (instead of error) when the file is empty.
• BigQuery copy operator -- copies a BQ table within or across projects without leaving Python.
• BigQuery User Defined Functions (UDFs) -- pass JS UDF definitions directly from the operator.
• BigQuery PEP 249 -- BigQuery hook now exposes a DB-API 2.0 interface, enabling use with standard SQL tooling.
• gcloud-based GCSHook -- an alternative GCS hook that delegates to the gcloud CLI, useful in environments where service-account JSON is impractical.
• Google Datastore Hook -- first-class hook for Cloud Datastore with persistent auth across the hook lifetime.
• Three-legged OAuth and domain-wide delegation -- broader auth support for Google Workspace (formerly G Suite) scenarios.
• Project specification in BigQuery hook methods -- lets you target a non-default GCP project at the method level rather than the connection level.

This matters if your team runs multi-project GCP environments -- you can now fan out to different projects within a single DAG without maintaining a separate Airflow connection per project.

One migration note: GCP API client dependencies are no longer force-installed. If your deployment relies on implicit transitive installation of google-api-python-client, add it to your pip requirements explicitly after upgrading.

What new operators and hooks does Airflow 1.7 introduce beyond Google Cloud?

Airflow 1.7 adds several operator primitives that have been on many teams' wish lists. The DockerOperator, SSHExecuteOperator, and the Qubole Data Services hook are the most production-relevant additions outside the GCP ecosystem.

• DockerOperator -- run arbitrary Docker containers as Airflow tasks. This opens up language-agnostic task execution without the overhead of a full CeleryExecutor worker stack.
• SSHExecuteOperator -- execute a command on a remote host over SSH directly from a task. Pairs with the SSH connection type now available in the web connection form.
• FTPSHook -- full FTP/FTPS connectivity, a gap that previously forced teams to fall back to BashOperator + curl.
• Qubole Data Services (QDS) operator and hook -- submit Hive, Hadoop, Pig, Spark, and Shell commands to Qubole clusters; Jinja template support included.
• PrestoToMySqlTransfer -- direct result-set transfer from Presto to MySQL, no intermediate file required.
• PigOperator stub -- lays the groundwork for Apache Pig integration.

The Presto hook also gets a meaningful defensive fix in 1.7: 503 errors from the Presto coordinator are now handled gracefully and no longer cause an uncontrolled crash. The old eval() call in the hook is removed -- a subtle but real security improvement.

How has the Airflow CLI improved in version 1.7?

The CLI in Airflow 1.7 becomes a significantly more capable debugging and scripting tool. Three changes stand out for daily use in production environments.

First, airflow trigger_dag now accepts --conf as a JSON string, allowing you to pass a runtime configuration dictionary to a DAG run without writing Python:

airflow trigger_dag my_dag_id --conf '{"date": "2016-03-01", "env": "prod"}'

Second, the airflow test command now accepts a JSON-formatted dictionary of parameters, making it straightforward to unit-test templated tasks with specific macro values from the command line.

Third, airflow render shows rendered templates for a task instance directly from the CLI -- previously you had to run the task or inspect the web UI to see how Jinja macros resolved. This saves significant debug time when chasing template rendering issues.

A new airflow dag_state command rounds out the additions, letting you query the state of a specific DAG run from a script without parsing web UI output or hitting the database directly. Most teams running on-call rotations will find this one immediately useful in runbooks.

Two new Jinja macros are also exposed in the task context: yesterday_ds_nodash and tomorrow_ds_nodash, filling a gap that previously forced teams to derive them manually in templated fields.

What authentication and monitoring improvements ship in Airflow 1.7?

Airflow 1.7 broadens authentication options and gives operators better runtime visibility through a StatsD abstraction and SLA miss callbacks.

On the authentication side:

• GitHub Enterprise OAuth -- teams on self-hosted GitHub can now authenticate Airflow users against their GitHub Enterprise instance without a custom plugin.
• LDAP search_scope -- a new search_scope configuration variable gives LDAP admins control over whether user lookups are scoped to a single OU or the entire directory tree. This is critical in large enterprise directories where a broad search produces ambiguous results.
• LDAP superuser and data-profiler roles -- LDAP group membership can now be mapped to Airflow's built-in superuser and data profiler permission levels.

On the observability side, a StatsD abstraction layer is introduced, giving Airflow a clean internal metrics interface. In practice this means you can route Airflow scheduler and task metrics to your existing StatsD-compatible pipeline (DataDog, Graphite, InfluxDB via telegraf, etc.) without patching the source. Configure it in airflow.cfg under the [scheduler] section.

The SLA miss callback is another production-ops win: DAGs can now specify a Python callable that fires whenever a task misses its SLA window. This matters if your SLA alerting previously relied on external polling of the Airflow REST API or scraping the web UI. The callback receives the DAG, task list, blocking task instances, and SLA miss records -- enough context to build a meaningful alert body without additional database queries.

Custom email backends round out this area, letting teams route alert emails through non-SMTP transports. SSL support for SMTP is also added for environments that require encrypted mail relay.