Consider dropping s390x support

[rikatz]

Ingress-nginx has supported both architectures for a long time now. We are considering dropping its support due to the following reasons:

• We cannot test against this architectures
• Compiling for those architectures takes a really long time
• We are not aware of people using it in production (newer versions)
• Some libraries that we use are not supported anymore on this versions (mainly luajit but also modsecurity and some other dependencies are not well tested)

Please comment below if you use it or not.

Target removal v1.10

Thanks

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rikatz]

@NeilHanlon I can see you -1. Can you please let us know your use case, and how are you using ingress-nginx on s390x?

Thanks

[skriesch]

SUSE Rancher is supporting ingress-nginx for s390x.

If I have understood it correctly in the Kubernetes news, you can build and test in the OBS (sponsored by SUSE) also based on s390x. Here is the link for the development project: https://build.opensuse.org/project/subprojects/isv:kubernetes

Say, if you need additional resources. IBM is providing additional mainframe resources for development and CI/CD for open source projects: https://openmainframeproject.org/news/developer-resources-for-linux-on-s390x/

[tao12345666333]

Thanks for your information, I will go get more details

[mfriesenegger]

Native s390x build resources that are permanent are available using the IBM LinuxONE Open Source Community Cloud. Please consider filling out the form if you are not using this free resource for open source projects.

[danwinship]

OpenShift internally uses haproxy, not nginx, for ingress, so I think we don't directly care about this, though it's possible we care because of things customers are doing with nginx. I'll ask around...

[skriesch]

@danwinship Do you provide continuously the Nginx Ingress Operator and also this one for s390x?

[rikatz]

folks, just making it clear, it is not just about resources but also about the burden. We have specific situations where we need to stick with old libraries (LuaJIT) because there's no recent patch for specific bugs on s390x as well.

Ingress NGINX compilation is still open, releasing specifically for s390x is the hard part. As an alternative, we could provide all the docs required to compile it for s390x (mainly the modifications, etc) but leave for the community and interested folks to do its compilation based on Ingress-NGINX releases.

Anyway, I'm still willing to listen more on cases (like if there's real usage vs just "we make it available"), stability reports, etc.

Thanks!

[skriesch]

openSUSE/SUSE and Fedora/RHEL/CentOS Stream have tested this patch for Luajit (also tested by IBM) and we are running successful builds of other software packages based on it: https://github.com/LuaJIT/LuaJIT/pull/631

You can give feedback to the Luajit people in this PR about your requirement, too. Then it "can" be merged perhaps. We are happy about every open source project supporting us there.

[rikatz]

@skriesch and this is one of the cases :) this issue has been opened for 3yrs now and we had to keep a specific compilation for s390x and luajit. I've seen similar problems with modsecurity and even other 3rd party C/C++ code

Because it is a very specific architecture it seems to be hard to get this kind of issue debugged and fixed.

If tomorrow there's another issue on some other part I'm afraid we will still have to wait for 3 more years and keep a detour on s390x compilation:/

[skriesch]

Use the Open Build Service. Then your problems will be solved. modsecurity is also available within openSUSE zSystems (openSUSE Tumbleweed for s390x). And here is Luajit. OBS is using all added patches by our Maintainers. You can choose your dependencies. The other Linux distributions are also supported. I can say only: "Try it!" :) Choose "openSUSE_Factory_zSystems" to receive the referenced packages as BuildRequires. Here is an example project with multiple Linux distributions for development by an upstream maintainer.

If you order VMs in the LinuxONE OSS Community Cloud (form link by @mfriesenegger above), you can use, build and develop based on it, too.

I and my openSUSE zSystems Team are providing also trainings regarding packaging and s390x. That is open for all. Next week is full with other topics, but we can plan something regarding OBS for 31st October 2023.

P.S. The single Linux distribution you can not build for on s390x in OBS is RHEL because of a missing available community Subscription for s390x (in the past that was possible as a minimum for x86). But you can use CentOS Stream instead.

[rposts]

@rikatz There are several open source projects on s390x arch that use ingress-nginx in one form or the other. I could think of strimzi-kafka-operator, Beats, Eclipse Che, etc. I would strongly vote to keep s390x support going as there may be other components using it.

[skriesch]

I and my openSUSE zSystems Team are providing also trainings regarding packaging and s390x. That is open for all. Next week is full with other topics, but we can plan something regarding OBS for 31st October 2023.

As mentioned, we want to give a multi-arch/multi-Linux packaging training with OBS. It will be 31st October 2023 at 19:00 (German time)/ 10:00 (Pacific time)/ 18:00 (Nigeria) / 14:00 (Brazil) at https://meet.opensuse.org/s390x

If you are looking for another time-zone, look for it based on German (Berlin) time, please. openSUSE zSystems is open for all with our trainings!

You will receive an introduction and overview, how the OBS is working, how to work with the tool osc on the command-line, how to build for different architectures and Linux distributions, the structure of a spec file and how to create a Submit Request.

Please like this post, if you want to attend. Then we can plan better.

[rikatz]

Hi, sorry for the delay, pretty busy week here. We have community meeting today, if you folks wanna have some time presenting it (or we can schedule for the next one)

I’m not sure I will be able to attend this one, but I’m happy to continue the conversation about this proposal (if someone is also at Kubecon and wanna chat, please ping me!!)

would it be an alternative that the LinuxOne maintain a supported build of it?

again, my bigger concern as of today is the specific branch/tags we need to track for s390x compilations (even long term we plan to drop luajit and other stuff and have a more “clean” NGINX compilation)

let’s chat!

[skriesch]

again, my bigger concern as of today is the specific branch/tags we need to track for s390x compilations (even long term we plan to drop luajit and other stuff and have a more “clean” NGINX compilation)

OBS is providing all, what is included into the provided Linux distributions, automatically. If you have got created a project configuration with required architectures and Linux distributions, you will receive automated builds for all requirements and afterwards the results. At the moment, a LinuxONE based on z15 is working in the background. SUSE wants to change back to z13 there. But "yes", you will receive supported builds for the different Linux distributions there. I expect, that Fedora, which will be released next month, will have also included the required patch for Luajit (equl to openSUSE Tumbleweed). If modsecurity is available for openSUSE for s390x, it should be also available for the next Fedora release. We are collaborating all together. Alternatively, feel free to use our projects for multiple Linux distributions to make it easier.

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[rikatz]

Hi all, just some news on it.

While we have been trying to keep the s390x binary generation working, we have hit an issue where it is not available on Alpine for s390x.

https://pkgs.alpinelinux.org/packages?name=opentelemetry-cpp&branch=edge&repo=&arch=s390x&maintainer=

This is a blocker for us to release the new NGINX v1.25 version.

The alternative we have considered was to compile opentelemetry-cpp from source but it was adding 1:50 hr to our build process, which right now is not acceptable because we run the image build as part of our CI when it is changed.

s390x binaries can still be generated from source and it is straitghforward to do it, but at this moment we have decided to not generate for ingress-nginx, until we have a viable fix for opentelemetry-cpp compilation.

An alternative (for maybe a next minor or patch release) is to re-add s390x but not use OTEL (remove it from the added packages if it is s390x arch, and from nginx compilation).

The issue here is that if a user of s390x decides to use otel, it will break their installation, but this should be a documented limitation as well.

Thanks for your patience, if someone is willing to help getting opentelemetry-cpp as part of Alpine s390x packages, please let me know once it is available!

[skriesch]

We have forwarded the topic to the OpenTelemetry Working Group@The Open Mainframe Project.

[rikatz]

Hey all,

I was able to build s390x images here with some mix between using alpine packages and compiling specific packages. Our compilation is still taking 1:40 hrs but we are targetting to improve it.

I will close this issue now as wont fix, and keep (for now) the generation of s390x releases.

Thanks