[FLINK-37775][python] Manage python development/test dependencies in pyproject.yaml

[autophagy]

What is the purpose of the change

Currently, development and testing dependencies for pyflink are scattered in a couple of locations. The dev dependencies are in dev/dev-requirements.txt, and the testing dependencies are scattered around the dev/lint-python.sh script. PEP 735 introduced support for dependency groups in pyproject.toml which, unlike [optional-dependencies] which are included in the distribution of the package, dependency groups are useful for defining named groups of dependencies for internal project use. Currently the latest versions of pip, uv, etc support this with pip install --group <group-name>.

This PR moves these dependencies into pyproject.toml and updates the docs, scripts, tests etc to use them.

Note: I'm unsure if moving the dev dependencies into here is the right thing to do, since people are still very used to pip install -r requirements.txt. I'd be open to keeping them in the requirements.txt file - but I do think keeping the testing dependencies in pyproject.toml instead of hardcoded in the lint-python.sh script is nicer.

Brief change log

• *Moved PyFlink's development and testing dependencies into pyproject.toml.

Verifying this change

This change is already covered by existing tests, such as the PyFlink integration and e2e tests.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: (no)
• The runtime per-record code paths (performance sensitive): (no)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
• The S3 file system connector: (no)

Documentation

• Does this pull request introduce a new feature? (no)
• If yes, how is the feature documented? (not applicable)

[flinkbot]

CI report:

• 1364cf443a2fdeee5235c419879f1d6241fe1c7f Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[dianfu]

@autophagy LGTM 👍. Could you rebase the PR to resolve the conflict?

[autophagy]

@dianfu Thanks for the review. I've rebased the PR and changed a few things:

• Changed the development group to dev since this is often better integrated into tooling (see https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies)
• Bumped the uv version being used so we can use uv pip install --group (https://github.com/astral-sh/uv/pull/13742)