Status of testing Providers that were prepared on May 01, 2024

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.8.0rc1

• [ ] fix(airbyte/hooks): add schema and port to prevent InvalidURL error (#38860): @josix Linked issues:
  • [ ] Linked Issue #37449: @henriquemeloo

Provider amazon: 8.21.0rc1

• [x] Added logging_config,snapstart,ephemeral_storage parameters to aws lambdacreatefunctionoperator (#39300): @gopidesupavan
• [ ] Fix bug in GlueJobOperator where consecutive runs fail when a local script file is used (#38960): @moritzsanne Linked issues:
  • [ ] Linked Issue #38959: @moritzsanne
• [ ] Update is_authorized_custom_view from auth manager to handle custom actions (#39167): @vincbeck Linked issues:
  • [ ] Linked Issue #39144: @jedcunningham
• [ ] Updated logic to allow AWS Batch Hook get_job_description retries to be more effective (#38998): @shahar1 Linked issues:
  • [ ] Linked Issue #37552: @evgenyslab
• [ ] Amazon Bedrock - Model Throughput Provisioning (#38850): @ferruzzi
• [ ] Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
• [ ] Allow importing the aws executors with a shorter path (#39093): @o-nikolas Linked issues:
  • [ ] Linked Issue #38524: @LipuFei
• [ ] Remove flag from AWS auth manager to use it (#39033): @vincbeck

Provider apache.beam: 5.7.0rc1

• [ ] Bugfix to correct GCSHook being called even when not required with BeamRunPythonPipelineOperator (#38716): @zstrathe Linked issues:
  • [ ] Linked Issue #38713: @zstrathe

Provider apache.livy: 3.8.0rc1

• [ ] Fix bug in LivyOperator when its trigger times out (#38916): @mateuslatrova
• [ ] Fix 'polling_interval' parameter docs in LivyOperator (#38979): @mateuslatrova
• [ ] Update docstring LivyOperator retry_args and deferrable docs (#39266): @boraberke Linked issues:
  • [ ] Linked Issue #29047: @sunank200

Provider apache.spark: 4.8.0rc2

• [ ] Rename SparkSubmitOperator argument queue as yarn_queue (#38852): @pateash Linked issues:
  • [ ] Linked Issue #38461: @djuarezg
  • [ ] Linked Issue #35911: @pateash

Provider cncf.kubernetes: 8.2.0rc1

• [ ] Add missing informative logs in KPO trigger until container has not finished (#37546): @pankajkoti
• [ ] Fix templated env_vars field in KubernetesPodOperator to allow for compatibility with XComArgs (#39139): @nyoungstudios
• [ ] Fix SparkKubernetesOperator when using initContainers (#38119): @ShelRoman
• [x] Refresh properties on KubernetesPodOperator when k8s fails due to token expiration (#39325): @paolo-moriello
• [ ] Remove unnecessary validation from cncf provider. (#39238): @VShkaberda
• [ ] Moves airflow import in deprecated pod_generator to local (#39062): @potiuk Linked issues:
  • [ ] Linked Issue #31442: @potiuk
• [ ] KPO xcom sidecar PodDefault usage (#38951): @jedcunningham

Provider common.sql: 1.13.0rc1

• [ ] Add sqlalchemy_url property to DbApiHook class (#38871): @rawwar Linked issues:
  • [ ] Linked Issue #38195: @Taragolis
• [ ] Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla
• [ ] Fix DbApiHook.insert_rows when rows is a generator (#38972): @vincbeck
• [ ] Fix update-common-sql-api-stubs pre-commit check (#38915): @Taragolis
• [ ] openlineage, snowflake: do not run external queries for Snowflake when (#39113): @mobuchowski

Provider databricks: 6.4.0rc1

• [x] Add DatabricksNotebookOperator (#39178): @pankajkoti
• [ ] Add notification settings paramaters (#39175): @SubhamSinghal
• [ ] [FEAT] raise exception with main notebook error in DatabricksRunNowDeferrableOperator (#39110): @gaurav7261

Provider datadog: 3.6.0rc1

• [ ] Add comment about versions updated by release manager (#37488): @potiuk

Provider dbt.cloud: 3.8.0rc1

• [ ] fix wrong payload set when reuse_existing_run set to True in DbtCloudRunJobOperator (#39271): @Lee-W
• [ ] migrate to dbt v3 api for project endpoints (#39214): @Lee-W

Provider fab: 1.1.0rc1

• [ ] Remove plugins permissions from Viewer role (#39254): @csp33 Linked issues:
  • [ ] Linked Issue #39218: @RocFang
• [ ] Update is_authorized_custom_view from auth manager to handle custom actions (#39167): @vincbeck Linked issues:
  • [ ] Linked Issue #39144: @jedcunningham

Provider google: 10.18.0rc1

• [ ] Add templated fields for Google Cloud Generative AI Operators (#39174): @CYarros10
• [ ] Add logic to handle on_kill for BigQueryInsertJobOperator when deferrable=True (#38912): @sunank200
• [ ] Create CloudComposerRunAirflowCLICommandOperator operator (#38965): @MaksYermak
• [ ] Add the deferrable mode to the Dataflow sensors (#37693): @e-galan
• [ ] Deferrable mode for Custom Training Job operators (#38584): @e-galan
• [ ] Enhancement for SSL-support in CloudSQLExecuteQueryOperator (#38894): @moiseenkov
• [ ] Create GKESuspendJobOperator and GKEResumeJobOperator operators (#38677): @MaksYermak
• [x] Add support for role arn for aws creds in storage transfer job operators (#38911): @wlinamchurch
• [ ] Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator (#39018): @e-galan
• [ ] Fix batching for BigQueryToPostgresOperator (#39233): @moiseenkov
• [ ] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. (#39230): @sunank200
• [ ] Fix GCSObjectExistenceSensor operator to return the same XCOM value in deferrable and non-deferrable mode (#39206): @VladaZakharova
• [ ] Bugfix BigQueryToMsSqlOperator (#39171): @moiseenkov
• [x] fix: add retry logic in case of google auth refresh credential error (#38961): @dondaum Linked issues:
  • [x] Linked Issue #38532: @nathadfield
• [ ] fix: BigQueryCheckOperator skip value and error check in deferrable mode (#38408): @kacpermuda Linked issues:
  • [ ] Linked Issue #37885: @kacpermuda
• [ ] [OpenLineage] Fix datasets in GCSDeleteObjectsOperator (#39059): @kacpermuda Linked issues:
  • [ ] Linked Issue #35838: @kacpermuda
• [ ] [OpenLineage] Use prefixes instead of file paths for datasets in GCSToGCSOperator (#39058): @kacpermuda Linked issues:
  • [ ] Linked Issue #31350: @mobuchowski
• [ ] [OpenLineage] Fix datasets in GCSTimeSpanFileTransformOperator (#39064): @kacpermuda Linked issues:
  • [ ] Linked Issue #35838: @kacpermuda
• [ ] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup (#39130): @sunank200
• [ ] Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
• [ ] Apply PROVIDE_PROJECT_ID mypy workaround across Google provider (#39129): @potiuk
• [ ] Handle ImportError for KubernetesDeleteJobOperator in google provider (#39036): @vatsrahul1001 Linked issues:
  • [ ] Linked Issue #37793: @MaksYermak

Provider hashicorp: 3.7.0rc2

• [x] Add assume_role_kwargs in hashicorp backend config (#39279): @pankajastro
• [x] Expose AWS IAM missing param in Hashicorp secret (#38536): @pankajastro

Provider http: 4.11.0rc1

• [ ] Allow trust env parameter to be defined in extra options of HTTP Connection (#39161): @dabla

Provider jdbc: 4.3.0rc1

• [ ] Suppress jaydebeapi.Error when setAutoCommit or getAutoCommit is unsupported by JDBC driver (#38707): @dabla
• [ ] Add stacklevel into the suppress_and_warn warning (#39263): @Taragolis

Provider microsoft.azure: 10.1.0rc1

• [ ] add dns_config and diagnostics parameters to AzureContainerInstancesOperator (#39156): @Kylea650 Linked issues:
  • [ ] Linked Issue #39149: @Kylea650
• [ ] Add stacklevel into the AzureSynapsePipelineHook deprecation warnings (#39192): @Taragolis
• [ ] Adding MSGraphOperator in Microsoft Azure provider (#38111): @dabla
• [ ] Pass through more parameters to AzureBlobFileSystem to mirror handlin… (#38747): @tomrutter Linked issues:
  • [ ] Linked Issue #38746: @tomrutter
• [ ] Implement run-method on KiotaRequestAdapterHook and move logic away from triggerer to hook (#39237): @dabla
• [ ] Fix: Only quote the keys of the query_parameters in MSGraphOperator (#39207): @dabla

Provider odbc: 4.6.0rc1

• [ ] Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla

Provider openai: 1.2.0rc1

• [x] OpenAI Chat & Assistant hook functions (#38736): @nathadfield
• [x] OpenAI Files & Vector Store Hooks (#39248): @nathadfield

Provider openlineage: 1.7.1rc1

• [ ] openlineage, snowflake: do not run external queries for Snowflake when (#39113): @mobuchowski

Provider pagerduty: 3.7.0rc2

• [ ] Cleanup PagerdutyNotifier init (#39145): @DavidTraina

Provider pinecone: 2.0.0rc1

• [ ] Pinecone provider support for pinecone-client>=3 (#37307): @rawwar Linked issues:
  • [ ] Linked Issue #37158: @Taragolis
• [ ] Bump ruff to 0.3.3 (#38240): @hussein-awala
• [ ] Add comment about versions updated by release manager (#37488): @potiuk

Provider postgres: 5.11.0rc1

• [x] Fix schema assigment in PostgresOperator (#39264): @eladkal
• [ ] Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla

Provider qdrant: 1.1.0rc1

• [x] Test connection in Qdrant provider (#39273): @Anush008

Provider salesforce: 5.7.0rc2

• [ ] update simple-salesforce type hints to support 1.12.6 (#39047): @hussein-awala

Provider slack: 8.7.0rc1

• [ ] Use upload_files_v2 Slack SDK method by default in Slack Operators (#39340): @Taragolis

Provider snowflake: 5.5.0rc1

• [ ] openlineage, snowflake: do not run external queries for Snowflake when (#39113): @mobuchowski

Provider teradata: 2.1.0rc1

• [ ] Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715): @dabla

All users involved in the PRs: @mobuchowski @josix @mateuslatrova @kacpermuda @dabla @Anush008 @jedcunningham @boraberke @pankajastro @Taragolis @gopidesupavan @hussein-awala @wlinamchurch @nathadfield @pankajkoti @potiuk @Kylea650

Committer

• [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[nathadfield]

@eladkal #39248 and #38736 tested and working correctly.

[dabla]

I know there is still an issue with the MSGraphSensor in that sense that it’s not working correctly in an async way and sometimes an event loop closed error can occur. This PR will fix it so dunno how to proceed for the msgraph part with this RC?

[wlinamchurch]

@eladkal https://github.com/apache/airflow/pull/38911 tested and working correctly.

[gopidesupavan]

@eladkal https://github.com/apache/airflow/pull/39300 tested and working correctly.

[Anush008]

Provider qdrant: 1.1.0rc1 https://github.com/apache/airflow/pull/39273

Tested. Works as expected.

[dondaum]

#38961 is good. ✅

[dabla]

Following 3 PR suffer from the same issues with the MSGraphSenor which has been fixed in #39304: #38111 #39237 #39207

[eladkal]

I know there is still an issue with the MSGraphSensor in that sense that it’s not working correctly in an async way and sometimes an event loop closed error can occur. This PR will fix it so dunno how to proceed for the msgraph part with this RC?

It's not merged. Only merged PR before release are included.

[dabla]

I know there is still an issue with the MSGraphSensor in that sense that it’s not working correctly in an async way and sometimes an event loop closed error can occur. This PR will fix it so dunno how to proceed for the msgraph part with this RC?

It's not merged. Only merged PR before release are included.

I know there is still an issue with the MSGraphSensor in that sense that it’s not working correctly in an async way and sometimes an event loop closed error can occur. This PR will fix it so dunno how to proceed for the msgraph part with this RC?

It's not merged. Only merged PR before release are included.

I know Elad I just wanted to make you guys aware of it.

[pankajastro]

Tested https://github.com/apache/airflow/pull/39279 and https://github.com/apache/airflow/pull/38536 looks good

[eladkal]

I know Elad I just wanted to make you guys aware of it.

If there is regression we can make RC2 then it will be included. What do you recommend ?

[paolo-moriello]

https://github.com/apache/airflow/pull/39325 tested 👍

[Kylea650]

Anybody with an Azure account able to test #39156? I can provide an example DAG I just don’t have access to my work Azure account right now and don’t want to block the release.

[rawwar]

Pineconepinecone: 2.0.0rc1

PR: #37307 is has bugs.

Fix is being implemented in #39365

[tomrutter]

Happy with #38747

[dabla]

I know Elad I just wanted to make you guys aware of it.

If there is regression we can make RC2 then it will be included. What do you recommend ?

Yes an RC2 would be great.

[pankajkoti]

Verified #39178 , works all fine

[boraberke]

#39266 looks good.

[zstrathe]

#38716 tested and verified #38713 is resolved

[nyoungstudios]

Tested https://github.com/apache/airflow/pull/39139 and looks good!

[Taragolis]

Tests my changes (most of them just cosmetic), look good

[Lee-W]

for https://github.com/apache/airflow/pull/38912, we might introduce a potential bug as commented in https://github.com/apache/airflow/issues/36090#issuecomment-2094972855

[pankajkoti]

for #38912, we might introduce a potential bug as commented in #36090 (comment)

Similar risk for #39130 and #39230

[eladkal]

Thank you everyone. Providers are released. Google, microsoft.azure and pinecone are excluded from this wave.

I invite everyone to help improve providers for the next release, a list of open issues can be found here.