public-datasets-pipelines icon indicating copy to clipboard operation
public-datasets-pipelines copied to clipboard

Published 20 hours ago • verified publisher icon GoogleCloudPlatform

Update dependency apache-airflow to v2.8.0 [SECURITY]

Open renovate-bot opened this issue 1 year ago • 1 comments

Mend Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
apache-airflow (source, changelog) ==2.2.5 -> ==2.8.0 age adoption passing confidence

GitHub Vulnerability Alerts

CVE-2023-25754

Privilege Context Switching Error vulnerability in Apache Software Foundation Apache Airflow. This issue affects Apache Airflow: before 2.6.0.

CVE-2022-46651

Apache Airflow, versions before 2.6.3, is affected by a vulnerability that allows an unauthorized actor to gain access to sensitive information in Connection edit view. This vulnerability is considered low since it requires someone with access to Connection resources specifically updating the connection to exploit it. Users should upgrade to version 2.6.3 or later which has removed the vulnerability.

CVE-2023-22887

Apache Airflow, versions before 2.6.3, is affected by a vulnerability that allows an attacker to perform unauthorized file access outside the intended directory structure by manipulating the run_id parameter. This vulnerability is considered low since it requires an authenticated user to exploit it. It is recommended to upgrade to a version that is not affected

CVE-2023-35908

Apache Airflow, versions before 2.6.3, is affected by a vulnerability that allows unauthorized read access to a DAG through the URL. It is recommended to upgrade to a version that is not affected

CVE-2023-36543

Apache Airflow, versions before 2.6.3, has a vulnerability where an authenticated user can use crafted input to make the current request hang. It is recommended to upgrade to a version that is not affected

CVE-2023-39508

Execution with Unnecessary Privileges, : Exposure of Sensitive Information to an Unauthorized Actor vulnerability in Apache Software Foundation Apache Airflow.The "Run Task" feature enables authenticated user to bypass some of the restrictions put in place. It allows to execute code in the webserver context as well as allows to bypas limitation of access the user has to certain DAGs. The "Run Task" feature is considered dangerous and it has been removed entirely in Airflow 2.6.0.

This issue affects Apache Airflow: before 2.6.0.

CVE-2023-40273

The session fixation vulnerability allowed the authenticated user to continue accessing Airflow webserver even after the password of the user has been reset by the admin - up until the expiry of the session of the user. Other than manually cleaning the session database (for database session backend), or changing the secure_key and restarting the webserver, there were no mechanisms to force-logout the user (and all other users with that).

With this fix implemented, when using the database session backend, the existing sessions of the user are invalidated when the password of the user is reset. When using the securecookie session backend, the sessions are NOT invalidated and still require changing the secure key and restarting the webserver (and logging out all other users), but the user resetting the password is informed about it with a flash message warning displayed in the UI. Documentation is also updated explaining this behaviour.

Users of Apache Airflow are advised to upgrade to version 2.7.0 or newer to mitigate the risk associated with this vulnerability.

CVE-2023-37379

Apache Airflow, in versions prior to 2.7.0, contains a security vulnerability that can be exploited by an authenticated user possessing Connection edit privileges. This vulnerability allows the user to access connection information and exploit the test connection feature by sending many requests, leading to a denial of service (DoS) condition on the server. Furthermore, malicious actors can leverage this vulnerability to establish harmful connections with the server.

Users of Apache Airflow are strongly advised to upgrade to version 2.7.0 or newer to mitigate the risk associated with this vulnerability. Additionally, administrators are encouraged to review and adjust user permissions to restrict access to sensitive functionalities, reducing the attack surface.

CVE-2023-39441

Apache Airflow SMTP Provider before 1.3.0, Apache Airflow IMAP Provider before 3.3.0, and Apache Airflow before 2.7.0 are affected by the Validation of OpenSSL Certificate vulnerability.

The default SSL context with SSL library did not check a server's X.509 certificate.  Instead, the code accepted any certificate, which could result in the disclosure of mail server credentials or mail contents when the client connects to an attacker in a MITM position.

Users are strongly advised to upgrade to Apache Airflow version 2.7.0 or newer, Apache Airflow IMAP Provider version 3.3.0 or newer, and Apache Airflow SMTP Provider version 1.3.0 or newer to mitigate the risk associated with this vulnerability

CVE-2023-40611

Apache Airflow, versions before 2.7.1, is affected by a vulnerability that allows authenticated and DAG-view authorized Users to modify some DAG run detail values when submitting notes. This could have them alter details such as configuration parameters, start date, etc.

Users should upgrade to version 2.7.1 or later which has removed the vulnerability.

CVE-2023-40712

Apache Airflow, versions before 2.7.1, is affected by a vulnerability that allows authenticated users who have access to see the task/dag in the UI, to craft a URL, which could lead to unmasking the secret configuration of the task that otherwise would be masked in the UI.

Users are strongly advised to upgrade to version 2.7.1 or later which has removed the vulnerability.

CVE-2023-42663

Apache Airflow, versions before 2.7.2, has a vulnerability that allows an authorized user with access to read specific DAGs only to read information about task instances in other DAGs. Users of Apache Airflow are advised to upgrade to version 2.7.2 or newer to mitigate the risk associated with this vulnerability.

CVE-2023-42792

Apache Airflow, in versions prior to 2.7.2, contains a security vulnerability that allows an authenticated user with limited access to some DAGs, to craft a request that could give the user write access to various DAG resources for DAGs that the user had no access to, thus, enabling the user to clear DAGs they shouldn't.

Users of Apache Airflow are strongly advised to upgrade to version 2.7.2 or newer to mitigate the risk associated with this vulnerability.

CVE-2023-42780

Apache Airflow, versions prior to 2.7.2, contains a security vulnerability that allows authenticated users of Airflow to list warnings for all DAGs, even if the user had no permission to see those DAGs. It would reveal the dag_ids and the stack-traces of import errors for those DAGs with import errors. Users of Apache Airflow are advised to upgrade to version 2.7.2 or newer to mitigate the risk associated with this vulnerability.

CVE-2023-47037

Apache Airflow, versions before 2.7.3, is affected by a vulnerability that allows authenticated and DAG-view authorized Users to modify some DAG run detail values when submitting notes. This could have them alter details such as configuration parameters, start date, etc.  Users should upgrade to version 2.7.3 or later which has removed the vulnerability.

CVE-2023-42781

Apache Airflow, versions before 2.7.3, has a vulnerability that allows an authorized user who has access to read specific DAGs only, to read information about task instances in other DAGs.  This is a different issue than CVE-2023-42663 but leading to similar outcome. Users of Apache Airflow are advised to upgrade to version 2.7.3 or newer to mitigate the risk associated with this vulnerability.

CVE-2023-50783

Apache Airflow, versions before 2.8.0, is affected by a vulnerability that allows an authenticated user without the variable edit permission, to update a variable. This flaw compromises the integrity of variable management, potentially leading to unauthorized data modification. Users are recommended to upgrade to 2.8.0, which fixes this issue.

CVE-2023-48291

Apache Airflow, in versions prior to 2.8.0, contains a security vulnerability that allows an authenticated user with limited access to some DAGs, to craft a request that could give the user write access to various DAG resources for DAGs that the user had no access to, thus, enabling the user to clear DAGs they shouldn't.

This is a missing fix for CVE-2023-42792 in Apache Airflow 2.7.2 

Users of Apache Airflow are strongly advised to upgrade to version 2.8.0 or newer to mitigate the risk associated with this vulnerability.

Release Notes

apache/airflow (apache-airflow)

v2.8.0

Compare Source

Significant Changes ^^^^^^^^^^^^^^^^^^^

  • Raw HTML code in DAG docs and DAG params descriptions is disabled by default

    To ensure that no malicious javascript can be injected with DAG descriptions or trigger UI forms by DAG authors a new parameter webserver.allow_raw_html_descriptions was added with default value of False. If you trust your DAG authors code and want to allow using raw HTML in DAG descriptions and params, you can restore the previous behavior by setting the configuration value to True.

    To ensure Airflow is secure by default, the raw HTML support in trigger UI has been super-seeded by markdown support via the description_md attribute. If you have been using description_html please migrate to description_md. The custom_html_form is now deprecated. (#​35460)

New Features """"""""""""

  • AIP-58: Add Airflow ObjectStore (AFS) (AIP-58 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-58+milestone%3A%22Airflow+2.8.0%22>_)
  • Add XCom tab to Grid (#​35719)
  • Add "literal" wrapper to disable field templating (#​35017)
  • Add task context logging feature to allow forwarding messages to task logs (#​32646, #​32693, #​35857)
  • Add Listener hooks for Datasets (#​34418, #​36247)
  • Allow override of navbar text color (#​35505)
  • Add lightweight serialization for deltalake tables (#​35462)
  • Add support for serialization of iceberg tables (#​35456)
  • prev_end_date_success method access (#​34528)
  • Add task parameter to set custom logger name (#​34964)
  • Add pyspark decorator (#​35247)
  • Add trigger as a valid option for the db clean command (#​34908)
  • Add decorators for external and venv python branching operators (#​35043)
  • Allow PythonVenvOperator using other index url (#​33017)
  • Add Python Virtualenv Operator Caching (#​33355)
  • Introduce a generic export for containerized executor logging (#​34903)
  • Add ability to clear downstream tis in List Task Instances view (#​34529)
  • Attribute clear_number to track DAG run being cleared (#​34126)
  • Add BranchPythonVirtualenvOperator (#​33356)
  • Allow PythonVenvOperator using other index url (#​33017)
  • Add CLI notification commands to providers (#​33116)
  • Use dropdown instead of buttons when there are more than 10 retries in log tab (#​36025)

Improvements """"""""""""

  • Add multiselect to run state in grid view (#​35403)
  • Fix warning message in Connection.get_hook in case of ImportError (#​36005)
  • Add processor_subdir to import_error table to handle multiple dag processors (#​35956)
  • Consolidate the call of change_state to fail or success in the core executors (#​35901)
  • Relax mandatory requirement for start_date when schedule=None (#​35356)
  • Use ExitStack to manage mutation of secrets_backend_list in dag.test (#​34620)
  • improved visibility of tasks in ActionModal for taskinstance (#​35810)
  • Create directories based on AIRFLOW_CONFIG path (#​35818)
  • Implements JSON-string connection representation generator (#​35723)
  • Move BaseOperatorLink into the separate module (#​35032)
  • Set mark_end_on_close after set_context (#​35761)
  • Move external logs links to top of react logs page (#​35668)
  • Change terminal mode to cbreak in execute_interactive and handle SIGINT (#​35602)
  • Make raw HTML descriptions configurable (#​35460)
  • Allow email field to be templated (#​35546)
  • Hide logical date and run id in trigger UI form (#​35284)
  • Improved instructions for adding dependencies in TaskFlow (#​35406)
  • Add optional exit code to list import errors (#​35378)
  • Limit query result on DB rather than client in synchronize_log_template function (#​35366)
  • Allow description to be passed in when using variables CLI (#​34791)
  • Allow optional defaults in required fields with manual triggered dags (#​31301)
  • Permitting airflow kerberos to run in different modes (#​35146)
  • Refactor commands to unify daemon context handling (#​34945)
  • Add extra fields to plugins endpoint (#​34913)
  • Add description to pools view (#​34862)
  • Move cli's Connection export and Variable export command print logic to a separate function (#​34647)
  • Extract and reuse get_kerberos_principle func from get_kerberos_principle (#​34936)
  • Change type annotation for BaseOperatorLink.operators (#​35003)
  • Optimise and migrate to SA2-compatible syntax for TaskReschedule (#​33720)
  • Consolidate the permissions name in SlaMissModelView (#​34949)
  • Add debug log saying what's being run to EventScheduler (#​34808)
  • Increase log reader stream loop sleep duration to 1 second (#​34789)
  • Resolve pydantic deprecation warnings re update_forward_refs (#​34657)
  • Unify mapped task group lookup logic (#​34637)
  • Allow filtering event logs by attributes (#​34417)
  • Make connection login and password TEXT (#​32815)
  • Ban import Dataset from airflow package in codebase (#​34610)
  • Use airflow.datasets.Dataset in examples and tests (#​34605)
  • Enhance task status visibility (#​34486)
  • Simplify DAG trigger UI (#​34567)
  • Ban import AirflowException from airflow (#​34512)
  • Add descriptions for airflow resource config parameters (#​34438)
  • Simplify trigger name expression (#​34356)
  • Move definition of Pod*Exceptions to pod_generator (#​34346)
  • Add deferred tasks to the cluster_activity view Pools Slots (#​34275)
  • heartbeat failure log message fix (#​34160)
  • Rename variables for dag runs (#​34049)
  • Clarify new_state in OpenAPI spec (#​34056)
  • Remove version top-level element from docker compose files (#​33831)
  • Remove generic trigger cancelled error log (#​33874)
  • Use NOT EXISTS subquery instead of tuple_not_in_condition (#​33527)
  • Allow context key args to not provide a default (#​33430)
  • Order triggers by - TI priority_weight when assign unassigned triggers (#​32318)
  • Add metric triggerer_heartbeat (#​33320)
  • Allow airflow variables export to print to stdout (#​33279)
  • Workaround failing deadlock when running backfill (#​32991)
  • add dag_run_ids and task_ids filter for the batch task instance API endpoint (#​32705)
  • Configurable health check threshold for triggerer (#​33089)
  • Rework provider manager to treat Airflow core hooks like other provider hooks (#​33051)
  • Ensure DAG-level references are filled on unmap (#​33083)
  • Affix webserver access_denied warning to be configurable (#​33022)
  • Add support for arrays of different data types in the Trigger Form UI (#​32734)
  • Add a mechanism to warn if executors override existing CLI commands (#​33423)

Bug Fixes """""""""

  • Account for change in UTC offset when calculating next schedule (#​35887)
  • Add read access to pools for viewer role (#​35352)
  • Fix gantt chart queued duration when queued_dttm is greater than start_date for deferred tasks (#​35984)
  • Avoid crushing container when directory is not found on rm (#​36050)
  • Update reset_user_sessions to work from either CLI or web (#​36056)
  • Fix UI Grid error when DAG has been removed. (#​36028)
  • Change Trigger UI to use HTTP POST in web ui (#​36026)
  • Fix airflow db shell needing an extra key press to exit (#​35982)
  • Change dag grid overscroll behaviour to auto (#​35717)
  • Run triggers inline with dag test (#​34642)
  • Add borderWidthRight to grid for Firefox scrollbar (#​35346)
  • Fix for infinite recursion due to secrets_masker (#​35048)
  • Fix write processor_subdir in serialized_dag table (#​35661)
  • Reload configuration for standalone dag file processor (#​35725)
  • Long custom operator name overflows in graph view (#​35382)
  • Add try_number to extra links query (#​35317)
  • Prevent assignment of non JSON serializable values to DagRun.conf dict (#​35096)
  • Numeric values in DAG details are incorrectly rendered as timestamps (#​35538)
  • Fix Scheduler and triggerer crashes in daemon mode when statsd metrics are enabled (#​35181)
  • Infinite UI redirection loop after deactivating an active user (#​35486)
  • Bug fix fetch_callback of Partial Subset DAG (#​35256)
  • Fix DagRun data interval for DeltaDataIntervalTimetable (#​35391)
  • Fix query in get_dag_by_pickle util function (#​35339)
  • Fix TriggerDagRunOperator failing to trigger subsequent runs when reset_dag_run=True (#​35429)
  • Fix weight_rule property type in mappedoperator (#​35257)
  • Bugfix/prevent concurrency with cached venv (#​35258)
  • Fix dag serialization (#​34042)
  • Fix py/url-redirection by replacing request.referrer by get_redirect() (#​34237)
  • Fix updating variables during variable imports (#​33932)
  • Use Literal from airflow.typing_compat in Airflow core (#​33821)
  • Always use Literal from typing_extensions (#​33794)

Miscellaneous """""""""""""

Doc Only Changes """"""""""""""""

  • Add support for tabs (and other UX components) to docs (#​36041)
  • Replace architecture diagram of Airflow with diagrams-generated one (#​36035)
  • Add the section describing the security model of DAG Author capabilities (#​36022)
  • Enhance docs for zombie tasks (#​35825)
  • Reflect drop/add support of DB Backends versions in documentation (#​35785)
  • More detail on mandatory task arguments (#​35740)
  • Indicate usage of the re2 regex engine in the .airflowignore documentation. (#​35663)
  • Update best-practices.rst (#​35692)
  • Update dag-run.rst to mention Airflow's support for extended cron syntax through croniter (#​35342)
  • Update webserver.rst to include information of supported OAuth2 providers (#​35237)
  • Add back dag_run to docs (#​35142)
  • Fix rst code block format (#​34708)
  • Add typing to concrete taskflow examples (#​33417)
  • Add concrete examples for accessing context variables from TaskFlow tasks (#​33296)
  • Fix links in security docs (#​33329)

v2.7.3

Compare Source

Significant Changes ^^^^^^^^^^^^^^^^^^^

No significant changes.

Bug Fixes """""""""

  • Fix pre-mature evaluation of tasks in mapped task group (#​34337)
  • Add TriggerRule missing value in rest API (#​35194)
  • Fix Scheduler crash looping when dagrun creation fails (#​35135)
  • Fix test connection with codemirror and extra (#​35122)
  • Fix usage of cron-descriptor since BC in v1.3.0 (#​34836)
  • Fix get_plugin_info for class based listeners. (#​35022)
  • Some improvements/fixes for dag_run and task_instance endpoints (#​34942)
  • Fix the dags count filter in webserver home page (#​34944)
  • Return only the TIs of the readable dags when ~ is provided as a dag_id (#​34939)
  • Fix triggerer thread crash in daemon mode (#​34931)
  • Fix wrong plugin schema (#​34858)
  • Use DAG timezone in TimeSensorAsync (#​33406)
  • Mark tasks with all_skipped trigger rule as skipped if any task is in upstream_failed state (#​34392)
  • Add read only validation to read only fields (#​33413)

Misc/Internal """""""""""""

  • Improve testing harness to separate DB and non-DB tests (#​35160, #​35333)
  • Add pytest db_test markers to our tests (#​35264)
  • Add pip caching for faster build (#​35026)
  • Upper bound pendulum requirement to <3.0 (#​35336)
  • Limit sentry_sdk to 1.33.0 (#​35298)
  • Fix subtle bug in mocking processor_agent in our tests (#​35221)
  • Bump @babel/traverse from 7.16.0 to 7.23.2 in /airflow/www (#​34988)
  • Bump undici from 5.19.1 to 5.26.3 in /airflow/www (#​34971)
  • Remove unused set from SchedulerJobRunner (#​34810)
  • Remove warning about max_tis per query > parallelism (#​34742)
  • Improve modules import in Airflow core by moving some of them into a type-checking block (#​33755)
  • Fix tests to respond to Python 3.12 handling of utcnow in sentry-sdk (#​34946)
  • Add connexion<3.0 upper bound (#​35218)
  • Limit Airflow to < 3.12 (#​35123)
  • update moto version (#​34938)
  • Limit WTForms to below 3.1.0 (#​34943)

Doc Only Changes """"""""""""""""

  • Fix variables substitution in Airflow Documentation (#​34462)
  • Added example for defaults in conn.extras (#​35165)
  • Update datasets.rst issue with running example code (#​35035)
  • Remove mysql-connector-python from recommended MySQL driver (#​34287)
  • Fix syntax error in task dependency set_downstream example (#​35075)
  • Update documentation to enable test connection (#​34905)
  • Update docs errors.rst - Mention sentry "transport" configuration option (#​34912)
  • Update dags.rst to put SubDag deprecation note right after the SubDag section heading (#​34925)
  • Add info on getting variables and config in custom secrets backend (#​34834)
  • Document BaseExecutor interface in more detail to help users in writing custom executors (#​34324)
  • Fix broken link to airflow_local_settings.py template (#​34826)
  • Fixes python_callable function assignment context kwargs example in params.rst (#​34759)
  • Add missing multiple_outputs=True param in the TaskFlow example (#​34812)
  • Remove extraneous '>' in provider section name (#​34813)
  • Fix imports in extra link documentation (#​34547)

v2.7.2

Compare Source

Significant Changes ^^^^^^^^^^^^^^^^^^^

No significant changes

Bug Fixes """""""""

  • Check if the lower of provided values are sensitives in config endpoint (#​34712)
  • Add support for ZoneInfo and generic UTC to fix datetime serialization (#​34683, #​34804)
  • Fix AttributeError: 'Select' object has no attribute 'count' during the airflow db migrate command (#​34348)
  • Make dry run optional for patch task instance (#​34568)
  • Fix non deterministic datetime deserialization (#​34492)
  • Use iterative loop to look for mapped parent (#​34622)
  • Fix is_parent_mapped value by checking if any of the parent taskgroup is mapped (#​34587)
  • Avoid top-level airflow import to avoid circular dependency (#​34586)
  • Add more exemptions to lengthy metric list (#​34531)
  • Fix dag warning endpoint permissions (#​34355)
  • Fix task instance access issue in the batch endpoint (#​34315)
  • Correcting wrong time showing in grid view (#​34179)
  • Fix www cluster_activity view not loading due to standaloneDagProcessor templating (#​34274)
  • Set loglevel=DEBUG in 'Not syncing DAG-level permissions' (#​34268)
  • Make param validation consistent for DAG validation and triggering (#​34248)
  • Ensure details panel is shown when any tab is selected (#​34136)
  • Fix issues related to access_control={} (#​34114)
  • Fix not found ab_user table in the CLI session (#​34120)
  • Fix FAB-related logging format interpolation (#​34139)
  • Fix query bug in next_run_datasets_summary endpoint (#​34143)
  • Fix for TaskGroup toggles for duplicated labels (#​34072)
  • Fix the required permissions to clear a TI from the UI (#​34123)
  • Reuse _run_task_session in mapped render_template_fields (#​33309)
  • Fix scheduler logic to plan new dag runs by ignoring manual runs (#​34027)
  • Add missing audit logs for Flask actions add, edit and delete (#​34090)
  • Hide Irrelevant Dag Processor from Cluster Activity Page (#​33611)
  • Remove infinite animation for pinwheel, spin for 1.5s (#​34020)
  • Restore rendering of provider configuration with version_added (#​34011)

Doc Only Changes """"""""""""""""

  • Clarify audit log permissions (#​34815)
  • Add explanation for Audit log users (#​34814)
  • Import AUTH_REMOTE_USER from FAB in WSGI middleware example (#​34721)
  • Add information about drop support MsSQL as DB Backend in the future (#​34375)
  • Document how to use the system's timezone database (#​34667)
  • Clarify what landing time means in doc (#​34608)
  • Fix screenshot in dynamic task mapping docs (#​34566)
  • Fix class reference in Public Interface documentation (#​34454)
  • Clarify var.value.get and var.json.get usage (#​34411)
  • Schedule default value description (#​34291)
  • Docs for triggered_dataset_event (#​34410)
  • Add DagRun events (#​34328)
  • Provide tabular overview about trigger form param types (#​34285)
  • Add link to Amazon Provider Configuration in Core documentation (#​34305)
  • Add "security infrastructure" paragraph to security model (#​34301)
  • Change links to SQLAlchemy 1.4 (#​34288)
  • Add SBOM entry in security documentation (#​34261)
  • Added more example code for XCom push and pull (#​34016)
  • Add state utils to Public Airflow Interface (#​34059)
  • Replace markdown style link with rst style link (#​33990)
  • Fix broken link to the "UPDATING.md" file (#​33583)

Misc/Internal """""""""""""

  • Update min-sqlalchemy version to account for latest features used (#​34293)
  • Fix SesssionExemptMixin spelling (#​34696)
  • Restrict astroid version < 3 (#​34658)
  • Fail dag test if defer without triggerer (#​34619)
  • Fix connections exported output (#​34640)
  • Don't run isort when creating new alembic migrations (#​34636)
  • Deprecate numeric type python version in PythonVirtualEnvOperator (#​34359)
  • Refactor os.path.splitext to Path.* (#​34352, #​33669)
  • Replace = by is for type comparison (#​33983)
  • Refactor integer division (#​34180)
  • Refactor: Simplify comparisons (#​34181)
  • Refactor: Simplify string generation (#​34118)
  • Replace unnecessary dict comprehension with dict() in core (#​33858)
  • Change "not all" to "any" for ease of readability (#​34259)
  • Replace assert by if...raise in code (#​34250, #​34249)
  • Move default timezone to except block (#​34245)
  • Combine similar if logic in core (#​33988)
  • Refactor: Consolidate import and usage of random (#​34108)
  • Consolidate importing of os.path.* (#​34060)
  • Replace sequence concatenation by unpacking in Airflow core (#​33934)
  • Refactor unneeded 'continue' jumps around the repo (#​33849, #​33845, #​33846, #​33848, #​33839, #​33844, #​33836, #​33842)
  • Remove [project] section from pyproject.toml (#​34014)
  • Move the try outside the loop when this is possible in Airflow core (#​33975)
  • Replace loop by any when looking for a positive value in core (#​33985)
  • Do not create lists we don't need (#​33519)
  • Remove useless string join from core (#​33969)
  • Add TCH001 and TCH002 rules to pre-commit to detect and move type checking modules (#​33865)
  • Add cancel_trigger_ids to to_cancel dequeue in batch (#​33944)
  • Avoid creating unnecessary list when parsing stats datadog tags (#​33943)
  • Replace dict.items by dict.values when key is not used in core (#​33940)
  • Replace lambdas with comprehensions (#​33745)
  • Improve modules import in Airflow core by some of them into a type-checking block (#​33755)
  • Refactor: remove unused state - SHUTDOWN (#​33746, #​34063, #​33893)
  • Refactor: Use in-place .sort() (#​33743)
  • Use literal dict instead of calling dict() in Airflow core (#​33762)
  • remove unnecessary map and rewrite it using list in Airflow core (#​33764)
  • Replace lambda by a def method in Airflow core (#​33758)
  • Replace type func by isinstance in fab_security manager (#​33760)
  • Replace single quotes by double quotes in all Airflow modules (#​33766)
  • Merge multiple isinstance calls for the same object in a single call (#​33767)
  • Use a single statement with multiple contexts instead of nested statements in core (#​33769)
  • Refactor: Use f-strings (#​33734, #​33455)
  • Refactor: Use random.choices (#​33631)
  • Use str.splitlines() to split lines (#​33592)
  • Refactor: Remove useless str() calls (#​33629)
  • Refactor: Improve detection of duplicates and list sorting (#​33675)
  • Simplify conditions on len() (#​33454)

v2.7.1

Compare Source

Significant Changes ^^^^^^^^^^^^^^^^^^^

CronTriggerTimetable is now less aggressive when trying to skip a run (#​33404) """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

When setting catchup=False, CronTriggerTimetable no longer skips a run if the scheduler does not query the timetable immediately after the previous run has been triggered.

This should not affect scheduling in most cases, but can change the behaviour if a DAG is paused-unpaused to manually skip a run. Previously, the timetable (with catchup=False) would only start a run after a DAG is unpaused, but with this change, the scheduler would try to look at little bit back to schedule the previous run that covers a part of the period when the DAG was paused. This means you will need to keep a DAG paused longer (namely, for the entire cron period to pass) to really skip a run.

Note that this is also the behaviour exhibited by various other cron-based scheduling tools, such as anacron.

conf.set() becomes case insensitive to match conf.get() behavior (#​33452) """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

Also, conf.get() will now break if used with non-string parameters.

conf.set(section, key, value) used to be case sensitive, i.e. conf.set("SECTION", "KEY", value) and conf.set("section", "key", value) were stored as two distinct configurations. This was inconsistent with the behavior of conf.get(section, key), which was always converting the section and key to lower case.

As a result, configuration options set with upper case characters in the section or key were unreachable. That's why we are now converting section and key to lower case in conf.set too.

We also changed a bit the behavior of conf.get(). It used to allow objects that are not strings in the section or key. Doing this will now result in an exception. For instance, conf.get("section", 123) needs to be replaced with conf.get("section", "123").

Bug Fixes """""""""

  • Ensure that tasks wait for running indirect setup (#​33903)
  • Respect "soft_fail" for core async sensors (#​33403)
  • Differentiate 0 and unset as a default param values (#​33965)
  • Raise 404 from Variable PATCH API if variable is not found (#​33885)
  • Fix MappedTaskGroup tasks not respecting upstream dependency (#​33732)
  • Add limit 1 if required first value from query result (#​33672)
  • Fix UI DAG counts including deleted DAGs (#​33778)
  • Fix cleaning zombie RESTARTING tasks (#​33706)
  • SECURITY_MANAGER_CLASS should be a reference to class, not a string (#​33690)
  • Add back get_url_for_login in security manager (#​33660)
  • Fix 2.7.0 db migration job errors (#​33652)
  • Set context inside templates (#​33645)
  • Treat dag-defined access_control as authoritative if defined (#​33632)
  • Bind engine before attempting to drop archive tables (#​33622)
  • Add a fallback in case no first name and last name are set (#​33617)
  • Sort data before groupby in TIS duration calculation (#​33535)
  • Stop adding values to rendered templates UI when there is no dagrun (#​33516)
  • Set strict to True when parsing dates in webserver views (#​33512)
  • Use dialect.name in custom SA types (#​33503)
  • Do not return ongoing dagrun when a end_date is less than utcnow (#​33488)
  • Fix a bug in formatDuration method (#​33486)
  • Make conf.set case insensitive (#​33452)
  • Allow timetable to slightly miss catchup cutoff (#​33404)
  • Respect soft_fail argument when poke is called (#​33401)
  • Create a new method used to resume the task in order to implement specific logic for operators (#​33424)
  • Fix DagFileProcessor interfering with dags outside its processor_subdir (#​33357)
  • Remove the unnecessary <br> text in Provider's view (#​33326)
  • Respect soft_fail argument when ExternalTaskSensor runs in deferrable mode (#​33196)
  • Fix handling of default value and serialization of Param class (#​33141)
  • Check if the dynamically-added index is in the table schema before adding (#​32731)
  • Fix rendering the mapped parameters when using expand_kwargs method (#​32272)
  • Fix dependencies for celery and opentelemetry for Python 3.8 (#​33579)

Misc/Internal """""""""""""

  • Bring back Pydantic 1 compatibility ([#​34081](https://togithub.com/apache/airfl

Configuration

📅 Schedule: Branch creation - "" (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Never, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

  • [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

renovate-bot avatar Jan 01 '24 17:01 renovate-bot

⚠ Artifact update problem

Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.

♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

  • any of the package files in this branch needs updating, or
  • the branch becomes conflicted, or
  • you click the rebase/retry checkbox if found above, or
  • you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:

File name: poetry.lock
Creating virtualenv cloud-datasets-02SzFaNK-py3.8 in /home/ubuntu/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...


Because cloud-datasets depends on apache-airflow (==2.8.0) which depends on sqlalchemy (>=1.4.28,<2.0), sqlalchemy is required.
So, because cloud-datasets depends on SQLAlchemy (==1.3.24), version solving failed.

forking-renovate[bot] avatar Jan 01 '24 17:01 forking-renovate[bot]