chore(deps): update dependency apache-airflow to v3 #223
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![renovate[bot]](https://avatars.githubusercontent.com/in/2740?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
==1.10.14->==3.0.0Warning
Some dependencies could not be looked up. Check the Dependency Dashboard for more information.
Release Notes
apache/airflow (apache-airflow)
v3.0.0Compare Source
We are proud to announce the General Availability of Apache Airflow 3.0 — the most significant release in the project's
history. This version introduces a service-oriented architecture, a stable DAG authoring interface, expanded support for
event-driven and ML workflows, and a fully modernized UI built on React. Airflow 3.0 reflects years of community
investment and lays the foundation for the next era of scalable, modular orchestration.
Highlights
^^^^^^^^^^
Service-Oriented Architecture: A new Task Execution API and
airflow api-serverenable task execution in remote environments with improved isolation and flexibility (AIP-72).Edge Executor: A new executor that supports distributed, event-driven, and edge-compute workflows (AIP-69), now generally available.
Stable Authoring Interface: DAG authors should now use the new
airflow.sdknamespace to import core DAG constructs like@dag,@task, andDAG.Scheduler-Managed Backfills: Backfills are now scheduled and tracked like regular DAG runs, with native UI and API support (AIP-78).
DAG Versioning: Airflow now tracks structural changes to DAGs over time, enabling inspection of historical DAG definitions via the UI and API (AIP-66).
Asset-Based Scheduling: The dataset model has been renamed and redesigned as assets, with a new
@assetdecorator and cleaner event-driven DAG definition (AIP-74, AIP-75).Support for ML and AI Workflows: DAGs can now run with
logical_date=None, enabling use cases such as model inference, hyperparameter tuning, and non-interval workflows (AIP-83).Removal of Legacy Features: SLAs, SubDAGs, DAG and Xcom pickling, and several internal context variables have been removed. Use the upgrade tools to detect deprecated usage.
Split CLI and API Changes: The CLI has been split into
airflowandairflowctl(AIP-81), and REST API now defaults tological_date=Nonewhen triggering a new DAG run.Modern React UI: A complete UI overhaul built on React and FastAPI includes version-aware views, backfill management, and improved DAG and task introspection (AIP-38, AIP-84).
Migration Tooling: Use ruff and airflow config update to validate DAGs and configurations. Upgrade requires Airflow 2.7 or later and Python 3.9–3.12.
Significant Changes
^^^^^^^^^^^^^^^^^^^
Airflow 3.0 introduces the most significant set of changes since the 2.0 release, including architectural shifts, new
execution models, and improvements to DAG authoring and scheduling.
Task Execution API & Task SDK (AIP-72)
""""""""""""""""""""""""""""""""""""""
Airflow now supports a service-oriented architecture, enabling tasks to be executed remotely via a new Task Execution
API. This API decouples task execution from the scheduler and introduces a stable contract for running tasks outside of
Airflow's traditional runtime environment.
To support this, Airflow introduces the Task SDK — a lightweight runtime environment for running Airflow tasks in
external systems such as containers, edge environments, or other runtimes. This lays the groundwork for
language-agnostic task execution and brings improved isolation, portability, and extensibility to Airflow-based
workflows.
Airflow 3.0 also introduces a new
airflow.sdknamespace that exposes the core authoring interfaces for defining DAGsand tasks. DAG authors should now import objects like
DAG,@dag, and@taskfromairflow.sdkrather thaninternal modules. This new namespace provides a stable, forward-compatible interface for DAG authoring across future
versions of Airflow.
Edge Executor (AIP-69)
""""""""""""""""""""""
Airflow 3.0 introduces the Edge Executor as a generally available feature, enabling execution of tasks in
distributed or remote compute environments. Designed for event-driven and edge-compute use cases, the Edge Executor
integrates with the Task Execution API to support task orchestration beyond the traditional Airflow runtime. This
advancement facilitates hybrid and cross-environment orchestration patterns, allowing task workers to operate closer to
data or application layers.
Scheduler-Managed Backfills (AIP-78)
""""""""""""""""""""""""""""""""""""
Backfills are now fully managed by the scheduler, rather than being launched as separate command-line jobs. This change
unifies backfill logic with regular DAG execution and ensures that backfill runs follow the same scheduling, versioning,
and observability models as other DAG runs.
Airflow 3.0 also introduces native UI and REST API support for initiating and monitoring backfills, making them more
accessible and easier to integrate into automated workflows. These improvements lay the foundation for smarter, safer
historical reprocessing — now available directly through the Airflow UI and API.
DAG Versioning (AIP-66)
"""""""""""""""""""""""
Airflow 3.0 introduces native DAG versioning. DAG structure changes (e.g., renamed tasks, dependency shifts) are now
tracked directly in the metadata database. This allows users to inspect historical DAG structures through the UI and API,
and lays the foundation for safer backfills, improved observability, and runtime-determined DAG logic.
React UI Rewrite (AIP-38, AIP-84)
"""""""""""""""""""""""""""""""""
Airflow 3.0 ships with a completely redesigned user interface built on React and FastAPI. This modern architecture
improves responsiveness, enables more consistent navigation across views, and unlocks new UI capabilities — including
support for DAG versioning, asset-centric DAG definitions, and more intuitive filtering and search.
The new UI replaces the legacy Flask-based frontend and introduces a foundation for future extensibility and community
contributions.
Asset-Based Scheduling & Terminology Alignment (AIP-74, AIP-75)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The concept of Datasets has been renamed to Assets, unifying terminology with common practices in the modern
data ecosystem. The internal model has also been reworked to better support future features like asset partitions and
validations.
The
@assetdecorator and related changes to the DAG parser enable clearer, asset-centric DAG definitions, allowingAirflow to more naturally support event-driven and data-aware scheduling patterns.
Unified Scheduling Field
""""""""""""""""""""""""
Airflow 3.0 removes the legacy
schedule_intervalandtimetableparameters. DAGs must now use the unifiedschedulefield for all time- and event-based scheduling logic. This simplifies DAG definition and improvesconsistency across scheduling paradigms.
Updated Scheduling Defaults
"""""""""""""""""""""""""""
Airflow 3.0 changes the default behavior for new DAGs by setting
catchup_by_default = Falsein the configurationfile. This means DAGs that do not explicitly set
catchup=...will no longer backfill missed intervals by default.This change reduces confusion for new users and better reflects the growing use of on-demand and event-driven workflows.
Restricted Metadata Database Access
"""""""""""""""""""""""""""""""""""
Task code can no longer directly access the metadata database. Interactions with DAG state, task history, or DAG runs
must be performed via the Airflow REST API or exposed context. This change improves architectural separation and enables
remote execution.
Future Logical Dates No Longer Supported
"""""""""""""""""""""""""""""""""""""""""
Airflow no longer supports triggering DAG runs with a logical date in the future. This change aligns with the logical
execution model and removes ambiguity in backfills and event-driven DAGs. Use
logical_date=Noneto trigger runs withthe current timestamp.
Context Behavior for Asset and Manually Triggered DAGs
""""""""""""""""""""""""""""""""""""""""""""""""""""""
For DAG runs triggered by an Asset event or through the REST API without specifying a
logical_date, Airflow now setslogical_date=Noneby default. These DAG runs do not have a data interval, and attempting to accessdata_interval_start,data_interval_end, orlogical_datefrom the task context will raise aKeyError.DAG authors should use
dag_run.logical_dateand perform appropriate checks or fallbacks if supporting multipletrigger types. This change improves consistency with event-driven semantics but may require updates to existing DAGs
that assume these values are always present.
Improved Callback Behavior
""""""""""""""""""""""""""
Airflow 3.0 refines task callback behavior to improve clarity and consistency. In particular,
on_success_callbackisno longer executed when a task is marked as
SKIPPED, aligning it more closely with expected semantics.Updated Default Configuration
"""""""""""""""""""""""""""""
Several default configuration values have been updated in Airflow 3.0 to better reflect modern usage patterns and
simplify onboarding:
catchup_by_defaultis now set toFalseby default. DAGs will not automatically backfill unless explicitly configured to do so.create_cron_data_intervalsis now set toFalseby default. As a result, cron expressions will be interpreted using theCronTriggerTimetableinstead of the legacyCronDataIntervalTimetable.SimpleAuthManageris now the defaultauth_manager. To continue using Flask AppBuilder-based authentication, install theapache-airflow-providers-flask-appbuilderprovider and explicitly setauth_manager = airflow.providers.fab.auth_manager.FabAuthManager.These changes represent the most significant evolution of the Airflow platform since the release of 2.0 — setting the
stage for more scalable, event-driven, and language-agnostic orchestration in the years ahead.
Executor & Scheduler Updates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3.0 introduces several important improvements and behavior changes in how DAGs and tasks are scheduled,
prioritized, and executed.
Standalone DAG Processor Required
"""""""""""""""""""""""""""""""""
Airflow 3.0 now requires the standalone DAG processor to parse DAGs. This dedicated process improves scheduler
performance, isolation, and observability. It also simplifies architecture by clearly separating DAG parsing from
scheduling logic. This change may affect custom deployments that previously used embedded DAG parsing.
Priority Weight Capped by Pool Slots
"""""""""""""""""""""""""""""""""""""
The
priority_weightvalue on a task is now capped by the number of available pool slots. This ensures that resourceavailability remains the primary constraint in task execution order, preventing high-priority tasks from starving others
when resource contention exists.
Teardown Task Handling During DAG Termination
"""""""""""""""""""""""""""""""""""""""""""""
Teardown tasks will now be executed even when a DAG run is terminated early. This ensures that cleanup logic is
respected, improving reliability for workflows that use teardown tasks to manage ephemeral infrastructure, temporary
files, or downstream notifications.
Improved Scheduler Fault Tolerance
""""""""""""""""""""""""""""""""""
Scheduler components now use
run_with_db_retriesto handle transient database issues more gracefully. This enhancesAirflow's fault tolerance in high-volume environments and reduces the likelihood of scheduler restarts due to temporary
database connection problems.
Mapped Task Stats Accuracy
"""""""""""""""""""""""""""
Airflow 3.0 fixes a bug that caused incorrect task statistics to be reported for dynamic task mapping. Stats now
accurately reflect the number of mapped task instances and their statuses, improving observability and debugging for
dynamic workflows.
DAG Authoring Enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3.0 includes several changes that improve consistency, clarity, and long-term stability for DAG authors.
Renamed Parameter:
fail_stop→fail_fast"""""""""""""""""""""""""""""""""""""""""""""""""
The DAG argument
fail_stophas been renamed tofail_fastfor improved clarity. This parameter controls whether aDAG run should immediately stop execution when a task fails. DAG authors should update any code referencing
fail_stopto use the new name.Context Cleanup and Parameter Removal
"""""""""""""""""""""""""""""""""""""
Several legacy context variables have been removed or may no longer be available in certain types of DAG runs,
including:
confexecution_datedag_run.external_triggerIn asset-triggered and manually triggered DAG runs with
logical_date=None, data interval fields such asdata_interval_startanddata_interval_endmay not be present in the task context. DAG authors should useexplicit references such as
dag_run.logical_dateand conditionally check for the presence of interval-related fieldswhere applicable.
Task Context Utilities Moved
""""""""""""""""""""""""""""
Internal task context functions such as
get_parsing_contexthave been moved to a more appropriate location (e.g.,airflow.models.taskcontext). DAG authors using these utilities directly should update import paths accordingly.Trigger Rule Restrictions
"""""""""""""""""""""""""
The
TriggerRule.ALWAYSrule can no longer be used with teardown tasks or tasks that are expected to honor upstreamdependency semantics. DAG authors should ensure that teardown logic is defined with the appropriate trigger rules for
consistent task resolution behavior.
Asset Aliases for Reusability
"""""""""""""""""""""""""""""
A new utility function,
create_asset_aliases(), allows DAG authors to define reusable aliases for frequentlyreferenced Assets. This improves modularity and reuse across DAG files and is particularly helpful for teams adopting
asset-centric DAGs.
Support for ML & AI Use Cases (AIP-83)
"""""""""""""""""""""""""""""""""""""""
Airflow 3.0 expands the types of DAGs that can be expressed by removing the constraint that each DAG run must correspond
to a unique data interval. This change, introduced in AIP-83, enables support for workflows that don't operate on a
fixed schedule — such as model training, hyperparameter tuning, and inference tasks.
These ML- and AI-oriented DAGs often run ad hoc, are triggered by external systems, or need to execute multiple times
with different parameters over the same dataset. By allowing multiple DAG runs with
logical_date=None, Airflow nowsupports these scenarios natively without requiring workarounds.
Config & Interface Changes
^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3.0 introduces several configuration and interface updates that improve consistency, clarify ownership of core
utilities, and remove legacy behaviors that were no longer aligned with modern usage patterns.
Default Value Handling
""""""""""""""""""""""
Airflow no longer silently updates configuration options that retain deprecated default values. Users are now required
to explicitly set any config values that differ from the current defaults. This change improves transparency and
prevents unintentional behavior changes during upgrades.
Refactored Config Defaults
"""""""""""""""""""""""""""
Several configuration defaults have changed in Airflow 3.0 to better reflect modern usage patterns:
catchup_by_defaultis nowFalse. DAGs will not backfill missed intervals unless explicitly configured to do so.create_cron_data_intervalsis nowFalse. Cron expressions are now interpreted using theCronTriggerTimetableinstead of the legacyCronDataIntervalTimetable. This change simplifies interval logic and aligns with the future direction of Airflow's scheduling system.Refactored Internal Utilities
"""""""""""""""""""""""""""""
Several core components have been moved to more intuitive or stable locations:
SecretsMaskerclass has been relocated toairflow.utils.secrets_masker.ObjectStoragePathutility previously located underairflow.iois now available viaairflow.utils.object_storage_path.These changes simplify imports and reflect broader efforts to stabilize utility interfaces across the Airflow codebase.
Improved
inlet_events,outlet_events, andtriggering_asset_events"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Asset event mappings in the task context are improved to better support asset use cases, including new features introduced in AIP-74.
Events of an asset or asset alias are now accessed directly by a concrete object to avoid ambiguity. Using a
strto access events isno longer supported. Use an
AssetorAssetAliasobject, orAsset.refto refer to an entity explicitly instead, such as::Alternatively, two helpers
for_assetandfor_asset_aliasare added as shortcuts::The internal representation of asset event triggers now also includes an explicit
urifield, simplifying traceability andaligning with the broader asset-aware execution model introduced in Airflow 3.0. DAG authors interacting directly with
inlet_eventsmay need to update logic that assumes the previous structure.Behaviour change in
xcom_pull"""""""""""""""""""""""""""""""""
Pulling without setting
task_ids:In Airflow 2, the
xcom_pull()method allowed pulling XComs by key without specifying task_ids, despite the fact that the underlyingDB model defines task_id as part of the XCom primary key. This created ambiguity: if two tasks pushed XComs with the same key,
xcom_pull()would pull whichever one happened to be first, leading to unpredictable behavior.Airflow 3 resolves this inconsistency by requiring
task_idswhen pulling by key. This change aligns with the task-scoped nature ofXComs as defined by the schema, ensuring predictable and consistent behavior.
DAG Authors should update their dags to use
task_idsif their dags usedxcom_pullwithouttask_idssuch as::kwargs["ti"].xcom_pull(key="key")
Should be updated to::
kwargs["ti"].xcom_pull(task_ids="task1", key="key")
Return Type Change for Single Task ID:
In Airflow 2, when using
xcom_pull()with a single task ID in a list (e.g.,task_ids=["task1"]), it would return aLazyXComSelectSequenceobject containing one value. In Airflow 3.0.0, this behavior was changed to return the value directly.
So, if you previously used:
.. code-block:: python
You would now get the value directly, rather than a sequence containing one value.
.. code-block:: python
The previous behaviour (returning list when passed a list) will be restored in Airflow 3.0.1 to maintain backward compatibility.
However, it is recommended to be explicit about your intentions when using
task_ids(after the fix in 3.0.1):task_ids="task1"task_ids=["task1"]This makes the code more explicit and easier to understand.
Removed Configuration Keys
"""""""""""""""""""""""""""
As part of the deprecation cleanup, several legacy configuration options have been removed. These include:
scheduler.allow_trigger_in_futurescheduler.use_job_schedulescheduler.use_local_tzUsers should review their
airflow.cfgfiles or use theairflow config lintcommand to identify outdated orremoved options.
Upgrade Tooling
""""""""""""""""
Airflow 3.0 includes improved support for upgrade validation. Use the following tools to proactively catch incompatible
configs or deprecated usage patterns:
airflow config lint: Identifies removed or invalid config keysruff check --select AIR30: Flags removed interfaces and common migration issuesCLI & API Changes
^^^^^^^^^^^^^^^^^
Airflow 3.0 introduces changes to both the CLI and REST API interfaces to better align with service-oriented deployments
and event-driven workflows.
Split CLI Architecture (AIP-81)
"""""""""""""""""""""""""""""""
The Airflow CLI has been split into two distinct interfaces:
airflowCLI now handles only local functionality (e.g.,airflow tasks test,airflow dags list).airflowctl, distributed via theapache-airflow-clientpackage.This change improves security and modularity for deployments that use Airflow in a distributed or API-first context.
REST API: DAG Trigger Behavior Updated
""""""""""""""""""""""""""""""""""""""
The behavior of the
POST /dags/{dag_id}/dagRunsendpoint has changed. If alogical_dateis not explicitlyprovided when triggering a DAG via the REST API, it now defaults to
None.This aligns with event-driven DAGs and manual runs in Airflow 3.0, but may break backward compatibility with scripts or
tools that previously relied on Airflow auto-generating a timestamped
logical_date.Removed CLI Flags and Commands
""""""""""""""""""""""""""""""
Several deprecated CLI arguments and commands that were marked for removal in earlier versions have now been cleaned up
in Airflow 3.0. Refer to the Deprecations & Removals section or run
airflow --helpto review the current set ofavailable commands and arguments.
Provider Refactor & Standardization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3.0 completes the migration of several core operators, sensors, and hooks into the new
apache-airflow-providers-standardpackage. This package now includes commonly used components such as:PythonOperatorBashOperatorEmailOperatorSimpleHttpOperatorShortCircuitOperatorThese operators were previously bundled inside
airflow-corebut are now treated as provider-managed components toimprove modularity, testability, and lifecycle independence.
This change enables more consistent versioning across providers and prepares Airflow for a future where all integrations
— including "standard" ones — follow the same interface model.
To maintain compatibility with existing DAGs, the
apache-airflow-providers-standardpackage is installable on bothAirflow 2.x and 3.x. Users upgrading from Airflow 2.x are encouraged to begin updating import paths and testing provider
installation in advance of the upgrade.
Legacy imports such as
airflow.operators.python.PythonOperatorare deprecated and will be removed soon. They should bereplaced with:
.. code-block:: python
UI & Usability Improvements
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3.0 introduces a modernized user experience that complements the new React-based UI architecture (see
Significant Changes). Several areas of the interface have been enhanced to improve visibility, consistency, and
navigability.
New Home Page
"""""""""""""
The Airflow Home page now provides a high-level operational overview of your environment. It includes health checks for
core components (Scheduler, Triggerer, DAG Processor), summary stats for DAG and task instance states, and a real-time
feed of asset-triggered events. This view helps users quickly identify pipeline health, recent activity, and potential
failures.
Unified DAG List View
""""""""""""""""""""""
The DAG List page has been refreshed with a cleaner layout and improved responsiveness. Users can browse DAGs by name,
tags, or owners. While full-text search has not yet been integrated, filters and navigation have been refined for
clarity in large deployments.
Version-Aware Graph and Grid Views
"""""""""""""""""""""""""""""""""""
The Graph and Grid views now display task information in the context of the DAG version that was used at runtime. This
improves traceability for DAGs that evolve over time and provides more accurate debugging of historical runs.
Expanded DAG Graph Visualization
""""""""""""""""""""""""""""""""
The Graph view now supports visualizing the full chain of asset and task dependencies, including assets consumed or
produced across DAG boundaries. This allows users to inspect upstream and downstream lineage in a unified view, making
it easier to trace data flows, debug triggering behavior, and understand conditional dependencies between assets and
tasks.
DAG Code View
"""""""""""""
The "Code" tab now displays the exact DAG source as parsed by the scheduler for the selected DAG version. This allows
users to inspect the precise code that was executed, even for historical runs, and helps debug issues related to
versioned DAG changes.
Improved Task Log Access
"""""""""""""""""""""""""
Task log access has been streamlined across views. Logs are now easier to access from both the Grid and Task Instance
pages, with cleaner formatting and reduced visual noise.
Enhanced Asset and Backfill Views
""""""""""""""""""""""""""""""""""
New UI components support asset-centric DAGs and backfill workflows:
These improvements make Airflow more accessible to operators, data engineers, and stakeholders working across both
time-based and event-driven workflows.
Deprecations & Removals
^^^^^^^^^^^^^^^^^^^^^^^^
A number of deprecated features, modules, and interfaces have been removed in Airflow 3.0, completing long-standing
migrations and cleanups.
Users are encouraged to review the following removals to ensure compatibility:
SubDag support has been removed entirely, including the
SubDagOperator, related CLI and API interfaces. TaskGroups are now the recommended alternative for nested DAG structures.SLAs have been removed: The legacy SLA feature, including SLA callbacks and metrics, has been removed. A more flexible replacement mechanism,
DeadlineAlerts, is planned for a future version of Airflow. Users who relied on SLA-based notifications should consider implementing custom alerting using task-level success/failure hooks or external monitoring integrations.Pickling support has been removed: All legacy features related to DAG pickling have been fully removed. This includes the
PickleDagCLI/API, as well as implicit behaviors aroundstore_serialized_dags = False. DAGs must now be serialized using the JSON-based serialization system. Ensure any custom Python objects used in DAGs are JSON-serializable.Context parameter cleanup: Several previously available context variables have been removed from the task execution context, including
conf,execution_date, anddag_run.external_trigger. These values are either no longer applicable or have been renamed (e.g., usedag_run.logical_dateinstead ofexecution_date). DAG authors should ensure that templated fields and Python callables do not reference these deprecated keys.Deprecated core imports have been fully removed. Any use of
airflow.operators.*,airflow.hooks.*, or similar legacy import paths should be updated to import from their respective providers.Configuration cleanup: Several legacy config options have been removed, including:
scheduler.allow_trigger_in_future: DAG runs can no longer be triggered with a future logical date. Uselogical_date=Noneinstead.scheduler.use_job_scheduleandscheduler.use_local_tzhave also been removed. These options were deprecated and no longer had any effect.Deprecated utility methods such as those in
airflow.utils.helpers,airflow.utils.process_utils, andairflow.utils.timezonehave been removed. Equivalent functionality can now be found in the standard Python library or Airflow provider modules.Removal of deprecated CLI flags and behavior: Several CLI entrypoints and arguments that were marked for removal in earlier versions have been cleaned up.
To assist with the upgrade, tools like
ruff(e.g., ruleAIR302) andairflow config lintcan help identifyobsolete imports and configuration keys. These utilities are recommended for locating and resolving common
incompatibilities during migration. Please see :doc:
Upgrade Guide <installation/upgrading_to_airflow3>for moreinformation.
Summary of Removed Features
"""""""""""""""""""""""""""
The following table summarizes user-facing features removed in 3.0 and their recommended replacements. Not all of these
are called out individually above.
+-------------------------------------------+----------------------------------------------------------+
| Feature | Replacement / Notes |
+===========================================+==========================================================+
| SubDagOperator / SubDAGs | Use TaskGroups |
+-------------------------------------------+----------------------------------------------------------+
| SLA callbacks / metrics | Deadline Alerts (planned post-3.0) |
+-------------------------------------------+----------------------------------------------------------+
| DAG Pickling | Use JSON serialization; pickling is no longer supported |
+-------------------------------------------+----------------------------------------------------------+
| Xcom Pickling | Use custom Xcom backend; pickling is no longer supported |
+-------------------------------------------+----------------------------------------------------------+
|
execution_datecontext var | Usedag_run.logical_date|+-------------------------------------------+----------------------------------------------------------+
|
confanddag_run.external_trigger| Removed from context; use DAG params ordag_runAPIs |+-------------------------------------------+----------------------------------------------------------+
| Core
EmailOperator| UseEmailOperatorfrom thesmtpprovider |+-------------------------------------------+----------------------------------------------------------+
|
none_failed_or_skippedrule | Usenone_failed_min_one_success|+-------------------------------------------+----------------------------------------------------------+
|
dummytrigger rule | Usealways|+-------------------------------------------+----------------------------------------------------------+
|
fail_stopargument | Usefail_fast|+-------------------------------------------+----------------------------------------------------------+
|
store_serialized_dags=False| DAGs are always serialized; config has no effect |+-------------------------------------------+----------------------------------------------------------+
| Deprecated core imports | Import from appropriate provider package |
+-------------------------------------------+----------------------------------------------------------+
| DebugExecutor | Use LocalExecutor for testing |
+-------------------------------------------+----------------------------------------------------------+
|
.airflowignoreregex | Uses glob syntax by default |+-------------------------------------------+----------------------------------------------------------+
Migration Tooling & Upgrade Process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Airflow 3 was designed with migration in mind. Many Airflow 2 DAGs will work without changes, especially if deprecation
warnings were addressed in earlier releases. To support the upgrade, Airflow 3 includes validation tools such as
ruffand
airflow config update, as well as a simplified startup model.For a step-by-step upgrade process, see the :doc:
Upgrade Guide <installation/upgrading_to_airflow3>.Minimum Supported Versions
"""""""""""""""""""""""""""
To upgrade to Airflow 3.0, you must be running Airflow 2.7 or later.
Airflow 3.0 supports the following Python versions:
Earlier versions of Airflow or Python are not supported due to architectural changes and updated dependency requirements.
DAG Compatibility Checks
"""""""""""""""""""""""""
Airflow now includes a Ruff-based linter with custom rules to detect DAG patterns and interfaces that are no longer
compatible with Airflow 3.0. These checks are packaged under the
AIR30xrule series. Example usage:.. code-block:: bash
These checks can automatically fix many common issues such as renamed arguments, removed imports, or legacy context
variable usage.
Configuration Migration
"""""""""""""""""""""""
Airflow 3.0 introduces a new utility to validate and upgrade your Airflow configuration file:
.. code-block:: bash
This utility detects removed or deprecated configuration options and, if desired, updates them in-place.
Additional validation is available via:
.. code-block:: bash
This command surfaces obsolete configuration keys and helps align your environment with Airflow 3.0 requirements.
Metadata Database Upgrade
"""""""""""""""""""""""""
As with previous major releases, the Airflow 3.0 upgrade includes schema changes to the metadata database. Before
upgrading, it is strongly recommended that you back up your database and optionally run:
.. code-block:: bash
to remove old task instance, log, or XCom data. To apply the new schema:
.. code-block:: bash
Startup Behavior Changes
"""""""""""""""""""""""""
Airflow components are now started explicitly. For example:
.. code-block:: bash
These changes reflect Airflow's new service-oriented architecture.
Resources
^^^^^^^^^
Upgrade Guide <installation/upgrading_to_airflow3>Airflow AIPs <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals>_Airflow 3.0 represents more than a year of collaboration across hundreds of contributors and dozens of organizations. We
thank everyone who helped shape this release through design discussions, code contributions, testing, documentation, and
community feedback. For full details, migration guidance, and upgrade best practices, refer to the official Upgrade
Guide and join the conversation on the Airflow dev and user mailing lists.
v2.10.5Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
Ensure teardown tasks are executed when DAG run is set to failed (#45530)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Previously when a DAG run was manually set to "failed" or to "success" state the terminal state was set to all tasks.
But this was a gap for cases when setup- and teardown tasks were defined: If teardown was used to clean-up infrastructure
or other resources, they were also skipped and thus resources could stay allocated.
As of now when setup tasks had been executed before and the DAG is manually set to "failed" or "success" then teardown
tasks are executed. Teardown tasks are skipped if the setup was also skipped.
As a side effect this means if the DAG contains teardown tasks, then the manual marking of DAG as "failed" or "success"
will need to keep the DAG in running state to ensure that teardown tasks will be scheduled. They would not be scheduled
if the DAG is directly set to "failed" or "success".
Bug Fixes
"""""""""
trigger_rule=TriggerRule.ALWAYSin a task-generated mapping within bare tasks (#44751)ONE_DONE) in a mapped task group (#44937)FileTaskHandleronly read from default executor (#46000)skip_ifandrun_ifdecorators before TaskFlow virtualenv tasks are run (#41832) (#45680)rendered_map_index(#45109) (#45122)max_form_parts,max_form_memory_size(#46243) (#45749)executesafeguard mechanism (#44646) (#46280)Miscellaneous
"""""""""""""
conffrom Task Context (#44993)v2.10.4Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
TaskInstance
priority_weightis capped in 32-bit signed integer ranges (#43611)"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Some database engines are limited to 32-bit integer values. As some users reported errors in
weight rolled-over to negative values, we decided to cap the value to the 32-bit integer. Even
if internally in python smaller or larger values to 64 bit are supported,
priority_weightiscapped and only storing values from -2147483648 to
2147483.Bug Fixes
^^^^^^^^^
trigger_rule="always"in a dynamic mapped task (#43810)trigger_rule=TriggerRule.ALWAYSin a task-generated mapping within bare tasks (#44751)Doc Only Changes
""""""""""""""""
Miscellaneous
"""""""""""""
v2.10.3Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes.
Bug Fixes
"""""""""
stringifiedobjects to UI via xcom if pickling is active (#42388) (#42486)selectinloadinstead ofjoinedload(#40487) (#42351)TrySelectorfor Mapped Tasks in Logs and Details Grid Panel (#43566)scheduler_loop_duration(#42886) (#43544)Miscellaneous
"""""""""""""
dompurifyfrom 2.2.9 to 2.5.6 in /airflow/www (#42263) (#42270)4.5.2(#43309) (#43318)Doc Only Changes
""""""""""""""""
v2.10.2Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes.
Bug Fixes
"""""""""
renderedTemplatesas keys to skipcamelCasing(#42206) (#42208)camelcasexcom entries (#42182) (#42187)Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Never, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.