[FLINK-34045][ci] Refactors docs build scripts and adds CI for pushing docs changes

[XComp]

What is the purpose of the change

The purpose of this PR is to consolidate the docs generation in a single script. That enables us to align the Hugo version that we rely on and reduce the amount of duplicated code.

Brief change log

• docs/docs_utils.sh collects all helper functions that are utilized by docs/build_docs.sh
• docs/build_docs.sh provides commands the following commands:
  • install-hugo installs the Hugo binaries locally with a fixed Hugo version
  • serve [--local|--docker] builds the docs and makes them accessible locally via localhost:1313 (analogously to what was done by ./docs/build_docs.sh)
  • build builds the docs locally without creating a webserver to access the docs
• Updates the README.md accordingly
• Creates Docs CI under ./.github/workflows/docs-ci.yml analogously to ./.github/workflows/ci.yml to trigger a CI run for docs changes (the ./docs path will be ignored in ci.yml and considered in docs-ci.yml
• Moves connector docs integration into ./docs/connector_docs.conf
• Makes the ./docs/build_docs.sh script not require to be called from within the ./docs/ folder
• Removes ./ci/tools/docs.sh script entirely

Verifying this change

• GitHub Actions workflow run

The subsections below give some guidance on how it can be tested locally: :warning: The exit codes are important for CI.

Serves docs via Docker

# executed from Flink's root folder
$ ./docs/build_docs.sh serve --docker

Build with error:

$ sed -i 's/config_file/cfg_file/g' docs/content/docs/deployment/config.md
$ ./docs/build_docs.sh serve --docker

Install Hugo locally

$ docker run -v $(pwd)/..:/checkout -it ubuntu:xenial
(docker) $ apt update && apt install -y curl git
(docker) $ which hugo && echo $? || echo $?
1 # confirms that no Hugo version is installed in the container
(docker) $ ./checkout/docs/build_docs.sh install-hugo && echo $? || echo $?
[INFO] Hugo (v0.110.0) installed to '/usr/local/bin'.
0 # confirms that the script didn't fail
(docker) $ ls /usr/local/bin/hugo
/usr/local/bin/hugo
(docker) $ hugo version
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=gohugoio

Runs the install logic with a failure:

# run with error
(docker) $ rm /usr/local/bin/hugo
(docker) $ which hugo && echo $? || echo $?
1 # confirms that no Hugo version is installed in the container
(docker) $ sed -i 's~https://github.com/gohugoio~https://github.com/wrong-url-asdf~g' docs/docs_utils.sh
(docker) $ ./checkout/docs/build_docs.sh install-hugo && echo $? || echo $?
[ERROR] Failed to download Hugo 0.110.0 binary using the following URL: https://github.com/wrong-url-asdf/hugo/releases/download/v0.110.0/hugo_extended_0.110.0_Linux-64bit.tar.gz
1 # indicates that the script returns a non-zero error code

Serves docs via a local Hugo installation

$ docker run -p1313:1313 -v $(pwd)/..:/checkout -it ubuntu:xenial
(docker) $ apt update && apt install -y curl git && ./checkout/docs/build_docs.sh install-hugo
(docker) $ which hugo && echo $? || echo $?
/usr/local/bin/hugo
0
(docker) $ ./checkout/docs/build_docs.sh serve --local
# docs are now accessible under https://localhost:1313

Serves with a failure:

(docker) $ sed -i 's/config_file/cfg_file/g' /checkout/docs/content/docs/deployment/config.md
# [...]
Start building sites … 
hugo v0.110.0-e32a493b7826d02763c3b79623952e625402b168+extended linux/amd64 BuildDate=2023-01-17T12:16:09Z VendorInfo=gohugoio
Error: Error building site: "/checkout/docs/content/docs/deployment/config.md:81:1": failed to extract shortcode: template for shortcode "cfg_file" not found
Built in 73 ms

Builds docs

$ docker run -p1313:1313 -v $(pwd)/..:/checkout -it ubuntu:xenial
(docker) $ apt update && apt install -y curl git && ./checkout/docs/build_docs.sh install-hugo
(docker) $ which hugo && echo $?
/usr/local/bin/hugo
0
(docker) $ ./checkout/docs/build_docs.sh build && echo $? || echo $?
# [...]
[INFO] Flink docs build from /checkout/docs into /checkout/docs/../target.
/
0

Build with failure:

(docker) $ sed -i 's/config_file/cfg_file/g' /checkout/docs/content/docs/deployment/config.md
(docker) $ ./checkout/docs/build_docs.sh build && echo $? || echo $?
# [...]
Error: Error building site: "/checkout/docs/content/docs/deployment/config.md:81:1": failed to extract shortcode: template for shortcode "cfg_file" not found
Total in 64 ms
255

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 6bb559e6066630995c7ac3758eff003cb9c13ac1 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[XComp]

Rebased the branch to most-recent master. There was a conflict reported due to the upgrade of the mongodb connector (v1.0 to v1.1) which is now addressed in this PR.

[XComp]

@flinkbot run azure