[FLINK-34487][ci] Adds Python Wheels nightly GHA workflow

[morazow]

What is the purpose of the change

Integrates Python Wheels build into the GitHub Actions (GHA) nightly workflow.

Brief change log

• Adds Python Wheels GHA workflow, template.python-wheels-ci.yml
• Integrates the Python Wheels workflow to the nightly GHA workflow

Verifying this change

This changes does not change code.

But the successful nightly GHA run should verify the change.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no

[flinkbot]

CI report:

• 38ac752c1e8abce741850b4b4b852311946b18c1 UNKNOWN
• 874064f5cd38098cfd55ac584c589fe7cf297d63 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[XComp]

fyi: I won't be responsive till the end of April at least

[morazow]

Thanks @XComp, I am going to address your suggestions. Please have a look once you are back

[morazow]

@flinkbot run azure

[morazow]

Thanks @XComp, going through your suggestions, I will address them, thanks 👍

No indeed, I am going to squash them, it should be single commit in the end

[morazow]

Thanks @XComp, I have addressed your findings, please have a look.

I used Python version 3.9 since it was in the Azure pipelines, let me know if we need to change it. Separated also the workflow stringify to another action, please have a look

[morazow]

Test run of Python wheel build: https://github.com/morazow/flink/actions/runs/9207285570

[morazow]

Artifacts from new nightly run:

[morazow]

Hey @XComp, @HuangXingBo ,

What do you think of the latest commit changes for migration?

This way both Linux & MacOS platform use similar build system, we also don't depend on bash script on Linux. All the wheel build requirements are defined in the pyproject.toml (python versions, architecture, etc) that are used for both platforms.

For Linux I had to add manylinux2014 version since manylinux1 does not support python3.10+ versions.

Please let me know what you think

[morazow]

Test run built wheel artifacts: https://github.com/morazow/flink/actions/runs/9224143298

[morazow]

Hey @XComp, could you please have look to the PR?

[HuangXingBo]

I downloaded the wheel package from https://github.com/morazow/flink/actions/runs/9224143298 and found that the wheel package for Linux is manylinux2014, not manylinux1, which may cause it to fail to install on some old machines. I used dev/glibc_version_fix.sh and patchelf in build-wheels.sh to solve this problem previously.

[morazow]

Thanks @HuangXingBo for the update!

I have changed the GitHub actions to run the auditwheel repair after installing patchelf in Linux. Could you please have another look?

You can find the build wheel artifacts for this change here: https://github.com/morazow/flink/actions/runs/9464510932

[HuangXingBo]

Thanks @morazow for the update. I downloaded the wheel package and found that the wheel package after repair is manylinux_2_5. It may be related to not using dev/glibc_version_fix.sh. If the version of glibc is too new, it is impossible to generate manylinux1 package.

[morazow]

Thanks @HuangXingBo for the feedback!

Should we still support manylinux1? It seems already end of life for some years

[HuangXingBo]

@morazow Since a lot of our dependencies are manyLinux2014, I think we can indeed build manyLinux2014.

[morazow]

Hello @HuangXingBo, thanks for the feedback!

Is the current state fine? It uses the manylinux2014 for Linux:

[tool.cibuildwheel.linux]
archs = ["x86_64"]
manylinux-x86_64-image = "manylinux2014"

Should I also keep the auditwheel steps?

[HuangXingBo]

@morazow I am not sure whether our wheel package can be manylinux2014 without auditwheel. We can try to remove auditwheel to see the effect

[morazow]

Hey @HuangXingBo,

I think one of the runs above was with manylinux2014 and without auditwheel step. But I will do another workflow run to create the wheels. I'll let you know

[morazow]

Hey @HuangXingBo,

I have run two builds, one is with default using auditwheel (together with patchelf) and another without auditwheel. Both on the manylinux2014 on Linux.

• With auditwheel: https://github.com/morazow/flink/actions/runs/9566314731
• Without auditwheel: https://github.com/morazow/flink/actions/runs/9566526379

I also updated the build workflow to make explicit some of the steps, previously it was running auditwheel by default on linux.

[HuangXingBo]

@morazow Thanks a lot for the experiment. It seems that we really need auditwheel, otherwise we can't generate manylinux packages.

[morazow]

Hey @HuangXingBo, thanks for looking into it 👍

I have added the auditwheel repair step and rebased. Please have a final look if all is fine.

[XComp]

Thanks for your help @HuangXingBo and @morazow for your persistence. :-) Code-wise, I'm fine after @HuangXingBo approved the functionality. 👍

@morazow can we get a final test run of the workflow in your fork (feel free to comment the extraneous parts of the workflow out in the debug commit)? We have to wait for the release branch cutting anyway.

[morazow]

Hey all,

Indeed, it took many conversations :)) But it was helpful collaborations! Thank a lot!

@XComp yes, this is the run with only Python wheel build: https://github.com/morazow/flink/actions/runs/9595245039. I will drop the comment commit and rebase once it is okay from your side also 👍

[XComp]

Thanks for the contribution, @morazow . I removed the extra commit and will go ahead with merging the PR. I guess we don't need to create backports for this kind of change because the nightlies are triggered from master. 🤔

But let's verify this with the nightly runs tomorrow.

[morazow]

Great, thanks @XComp, @HuangXingBo for the help 👍