setup-python icon indicating copy to clipboard operation
setup-python copied to clipboard

Caching for Graal

Open masklinn opened this issue 1 year ago • 1 comments

I'm not sure how the actions/setup-python caching is supposed to work, but every graal action indicates that it's "not found in the local cache", leading to a 30~35s overhead over cpython and pypy builds every time, contributing to the cost of graal jobs quite significantly (which adds onto graal's own issues).

masklinn avatar Oct 07 '24 15:10 masklinn

Hi @masklinn, Thank you for the feature request. We'll investigate it and come back to you with the details!

gowridurgad avatar Oct 08 '24 03:10 gowridurgad

Hello @masklinn, The actions/setup-python caching currently does not apply to GraalVM runtimes, which is why you're seeing repeated downloads and the associated overhead. Since GraalVM support was removed from hosted runners starting April 2023 (#7338), it is no longer pre-installed and must be fetched each time unless cached explicitly.

To mitigate this, you could consider:

Using setup-graalvm directly, which offers more granular control and may support caching improvements in the future.

Implementing a custom cache strategy for downloaded GraalVM builds using actions/cache.

We understand this overhead adds up, especially for CI jobs, and we’ve noted this feedback as a potential optimization area. Please let us know in case of any further support needed

aparnajyothi-y avatar Apr 28 '25 13:04 aparnajyothi-y

Using setup-graalvm directly, which offers more granular control and may support caching improvements in the future.

It's unclear to me that setup-graalvm can set up graalpython? I tried adding it to a test workflow (mostly to see if that would work around #880) and python kept resolving to whatever the distribution had, and trying to add setup-python afterwards didn't seem to have it pick up anything.

So I don't think graalvm would help? Which makes sense, graalpython is built on graalvm but is not part of the graalvm project itself, and graalpython's own readme goes through setup-python.

masklinn avatar May 04 '25 19:05 masklinn

Hi @masklinn, Thanks for the follow-up and for sharing your insights. You're absolutely right, setup-graalvm does not install or configure GraalPython in a way that setup-python can recognize. While GraalPython is built on GraalVM, it isn’t directly bundled with core GraalVM distributions or managed by either setup-graalvm or setup-python.

To clarify my earlier comment: the actions/setup-python caching mechanism does not apply to GraalPython or any GraalVM-based Python runtimes. Since GraalVM support was removed from GitHub-hosted runners in April 2023, it is no longer pre-installed. While setup-graalvm allows for manual GraalVM installation, setting up GraalPython still requires custom steps outside of these actions. Following further discussions with the runner-images team, there are currently no plans to add caching support for GraalPython, primarily due to memory and storage constraints on hosted runners. These limitations prevent us from pre-installing or maintaining GraalVM-related assets in the standard tool cache.

To mitigate the overhead in the meantime, you might consider:

Using setup-graalvm for manual installation of GraalVM (noting that GraalPython setup remains manual).

Implementing a custom caching layer with actions/cache to persist GraalVM and GraalPython artifacts between runs.

We recognize that repeated downloads affect CI efficiency, and while GraalPython isn’t officially supported today, your feedback is invaluable. We’ll continue monitoring demand and reassess feasibility if infrastructure constraints evolve.

Thanks again for highlighting this and for contributing. Please feel free to reach us in case of any concerns/clarifications needed.

aparnajyothi-y avatar Jun 13 '25 05:06 aparnajyothi-y

We recognize that repeated downloads affect CI efficiency, and while GraalPython isn’t officially supported today, your feedback is invaluable. We’ll continue monitoring demand and reassess feasibility if infrastructure constraints evolve.

I'll be completely honest it's for an OSS project which only has emergencies if I do something stupid and cut a broken release, so I'm not majorly affected personally.

masklinn avatar Jun 13 '25 17:06 masklinn

@aparnajyothi-y setup-graalvm and the removal of GraalVM support have nothing to do with GraalPy and setup-python. Absolutely nothing. There is no requirement to have GraalVM when you run GraalPy, GraalPy ships a standalone, native Python implementation in a tar.gz just like CPython and PyPy do.

And your comment "GraalPython isn’t officially supported today" is also misleading, since setup-python does have support for graalpy, see https://github.com/actions/setup-python/pull/694, https://github.com/actions/setup-python/pull/741, https://github.com/actions/setup-python/pull/773

timfel avatar Jun 23 '25 07:06 timfel

Hello @timfel, You're right in noting that GraalPy itself ships as a standalone Python implementation and doesn't strictly require GraalVM. I appreciate you referencing #694, #741, and #773, which indeed introduced foundational support for GraalPy in setup-python.

To clarify our earlier message: when we mentioned that GraalPython (GraalPy) “isn’t officially supported,” we were referring specifically to the level of support and integration parity with CPython and PyPy. While GraalPy is recognized and installable through setup-python, it is not available via the hosted tool-cache and not pre-installed on runner images. This limitation stems from the fact that GraalVM support was removed from GitHub-hosted runners starting April 2023 (#7338), meaning it must now be fetched on each use unless explicitly cached by the workflow.

This has led to issues like #954, where users expected the same seamless setup and caching experience as with other runtimes. So while GraalPy support exists in the action, it’s not yet at a fully supported parity level.

That said, we’re definitely aware of the growing interest in GraalPy, and we’ll continue evaluating deeper integration based on user demand and infrastructure feasibility.

Please feel free to reach out in case further concerns or clarifications are required.

aparnajyothi-y avatar Jul 11 '25 05:07 aparnajyothi-y

This limitation stems from the fact that GraalVM support was removed from GitHub-hosted runners starting April 2023 (#7338), meaning it must now be fetched on each use unless explicitly cached by the workflow.

That is just not true. There never was any caching for GraalPy as setup-python is concerned, and GraalVM being available in runners or not has no bearing on caching GraalPy, these are entirely different things.

timfel avatar Jul 12 '25 13:07 timfel

Hi @timfel, Our earlier reference to the removal of GraalVM support from GitHub-hosted runners (#7338) was intended to offer broader context on the evolution of runtime availability. That said, we understand that GraalPy follows a separate installation and packaging path.

While GraalPy is not currently available via the tool cache like CPython or PyPy, we recommend reaching out to the runner-images team directly to explore any potential options or to share specific use cases. This can help inform future planning and highlight any blockers tied to current runner configurations. Please don’t hesitate to reach out with any further questions or feedback.

aparnajyothi-y avatar Jul 22 '25 05:07 aparnajyothi-y

Hi @timfel, You’re correct, GraalPy has never been cached in runner-images the way CPython or PyPy are, and we’re aware there’s no caching support at present.

While GraalPy isn’t yet available via the tool cache like pypy and python, we recommend reaching out to the runner-images team to share your use case or explore possible options. This can help inform future planning and surface any blockers in current runner configurations.

Please feel free to reach out with any further questions or feedback.

aparnajyothi-y avatar Aug 14 '25 13:08 aparnajyothi-y

Hello @timfel, we are proceeding to close this issue as GraalPy support is already available through setup-python. However, please note that GraalPy has never been cached in runner-images and is not available via the tool cache like CPython or PyPy. For deeper integration, we recommend reaching out to the runner-images team to explore potential options for tool-cache support.

aparnajyothi-y avatar Oct 31 '25 03:10 aparnajyothi-y