Amazon Bedrock - Knowledge Bases and Data Sources

[ferruzzi]

Adds hooks, operators, sensors, triggers, and waiters to enable support for Amazon Bedrock Knowledge Bases. This includes Bedrock Data Sources and Ingestion Jobs, and some light additions (a thin hook, and a waiter/sensor/trigger set) for Amazon OpenSearch Serverless which are needed to make the system test work end-to-end.

System test and unit tests are included, as well as docs. Static checks, unit tests, and build-docs all pass locally.

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[vincbeck]

Not a big deal but you could have split this PR in 2; Bedrock and Opensearch. Especially because you do not mention opensearch in the PR title.

[ferruzzi]

I'm really not sure what that failing static check is trying to tell me. Wouldn't mind a hand on that if someone knows what it means.

[vincbeck]

I'm really not sure what that failing static check is trying to tell me. Wouldn't mind a hand on that if someone knows what it means.

I think, it tries to tell you that you created a doc (opensearchserverless.rst) that you do not reference anywhere. Adding a how-to-guide section in provider.yaml which references the new doc should solve it

[ferruzzi]

Adding a how-to-guide section in provider.yaml which references the new doc should solve it

I SEE! Some of the integrations list a how-to and others don't, I'll add that in and see if that fixes it.

I'll have the missing deferrable implementations up some time later today, I want to make sure I run them through manual testing which takes a bit of time per run.

[ferruzzi]

Alright. @vincbeck I believe I have address the concerns you raised

I just ran the system test end to end three times:

default_args={deferrable=True, wait_for_completion=False}

default_args={deferrable=False, wait_for_completion=True}

default_args={deferrable=False, wait_for_completion=False}

and it passes all three ways.

While I was at it, I was pretty behind so I also rebased. Sorry for the force push.

I'll have to sort out that failing static check on Monday, it looks like maybe I have a boto call in top-level code somewhere in the system test, perhaps? Not sure.

[ferruzzi]

Alright, got all the tests passing and rebased since it's been a few days. That was an adventure, sorry about that.

[ferruzzi]

how could the change in 6d5cdd2 have broken three test which were all passing before that commit? It's copy and pasted dag declaration from the rest of our system tests.

[ferruzzi]

Assuming the CI goes green, I'm done.

@vincbeck - you may want to have a peek at the changes since you approved, or not. @Taragolis - I ended up doing the generic base sensor your way and it worked perfectly. Thanks!