fix: Astroid 4 breaks docs building

[rich7420]

related: #56420

This PR switch to apidoc+autodoc only for provider docs when Astroid ≥ 4 and everything else still uses AutoAPI. On Sphinx startup, we run sphinx-apidoc to generate _api, sidestepping Astroid 4 vs AutoAPI ignore quirks. This PR changed the ignore list (exclude get_provider_info.py, version_compat.py, etc.) that aren’t useful for API and can break parsing. Also This updated Airbyte docs: Python API now links to _api/modules; System tests is a note instead of a hard toctree, avoiding dead links. W#ithout using version pinning or downgrades, once AutoAPI supports Astroid 4, we can just remove the conditional and revert easily.

test

uv run --group docs build-docs --docs-only

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[kaxil]

Are the docs & the elements visible though -- similar to https://airflow.apache.org/docs/task-sdk/stable/api.html?

[rich7420]

Are the docs & the elements visible though -- similar to https://airflow.apache.org/docs/task-sdk/stable/api.html?

Good point! This PR only affects provider docs. Task SDK docs use different config (autoapi_generate_api_docs = False) so they should still work.

[kaxil]

Good point! This PR only affects provider docs. Task SDK docs use different config (autoapi_generate_api_docs = False) so they should still work.

You need to confirm it works as it is important before we review the PR

[rich7420]

@kaxil thanks for your reminding. I've verified both docs are working. my changes:

cd providers/airbyte && uv run --group docs build-docs --docs-only --clean-build

unchanged:

cd task-sdk && uv run --group docs build-docs --docs-only --clean-build

Both generated complete API docs with full navigation, working links, and proper class/method documentation - similar structure to the Task SDK API page you referenced. My PR only touches provider docs - Task SDK uses different config (autoapi_generate_api_docs = False) so it's unaffected.

[potiuk]

I am afraid also "airflow-core" will need to be updated and all providers (Similarly as airbyte) there are also some consistency checks failing - with system_tests not being detected - all those problems should be fixed and the CI should get green before merging it.

[potiuk]

Needs conflict resolution :)

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[rich7420]

@kaxil , @potiuk sorry for updating so late. I found sphinx-autoapi updated in 3.6.1 to support Astroid 4.

[potiuk]

No worries. But seems that there is something changed in how autoapi treats some of the generated files:

2025-12-12T08:03:38.6691385Z apache-airflow-providers-vertica                             [1m[[0m91m/opt/airflow/providers/vertica/docs/index.rst:[1;36m47[0m: WARNING: toctree contains reference to nonexisting document [32m'_api/tests/system/vertica/index'[0m [1m[[0m[1;36m39[0m;[1;36m49[0m;00m

Or the files are generated differently ?