uv icon indicating copy to clipboard operation
uv copied to clipboard

Suggestion: `uv bundle`, `uv build --release` or similar to create a contained executable a la pyinstaller, py2exe

Open matterhorn103 opened this issue 1 year ago • 15 comments
trafficstars

I regularly use pyinstaller for a project, and the frequency of questions about it - or the task more generally of "compiling" an executable for distribution - around the web indicates to me that it's widely popular.

It would be incredibly cool if uv offered a uv bundle command (or whatever name) that essentially did what pyinstaller does and bundles together everything to afford an executable that contains the dependencies and interpreter and can be distributed to users as-is. And I think this falls into the realm of what things a "Cargo for Python" might be expected to be able to do.

I assume that pyinstaller, py2exe etc. are complex projects and implementing such functionality would be not trivial in the slightest, so I would have thought it would be a case of using pyinstaller itself in the background like the way that build backends are used.

Presumably there will one day be a uv build command but I assume that will be similar to rye build in that it will create an sdist or wheel. Maybe there could be a uv build --executable option, or uv build --release (by analogy to cargo build --release)?

matterhorn103 avatar Aug 05 '24 23:08 matterhorn103

Some relevant links that I'm adding to keep track of them for myself as much as anyone else:

  • It seems someone asked for the same in Rye previously
  • ofek's PyApp and Gregory Szorc's (indygreg) now-dead PyOxidizer are other, more modern executable generators that I didn't know about until now; both seem simpler than pyinstaller, and both are written in Rust
  • Gregory Szorc once did a comparison of the various alternatives to PyOxidizer
  • One missing from that list, presumably because it's not generally applicable, is the pyside6-deploy tool, but it uses Nuitka in the background in any case
  • Hatch seems to have implemented essentially this functionality, as part of a Hatch plugin that uses PyApp, and it is invoked there by having "binary" as a build target

matterhorn103 avatar Aug 06 '24 00:08 matterhorn103

To add… despite its popularity, pyinstaller suffers from slowness because has to unzip all the dependencies and the interpreter when the executable is launched. It offers users the option to display a splash image to distract them from the perceived slowness.

chrisrodrigue avatar Aug 06 '24 09:08 chrisrodrigue

Python to Rust transpilation would be pretty cool and there’s at least one project out there working on this:

Imagine writing idiomatic Python, transpiling it to Rust and compiling a native executable. My mind would be blown.

I don’t know how one could gracefully handle dependencies though. A lot of Python libs incorporate C extensions. Dealing with those would be extremely difficult, but there’s a DARPA program called TRACTOR working on that.

chrisrodrigue avatar Aug 06 '24 10:08 chrisrodrigue

An equivalent of pyinstaller that actually understands standard pyproject.toml-driven projects and console/GUI script entrypoints (instead of being fixated on script development, and overcompensating with its dependency detection magic) would be huge.

(somewhat related: #5653)

paveldikov avatar Aug 06 '24 14:08 paveldikov

I want to give a mention to Nuitka which is serving me well. It's as straightforward as running

uv add --dev nuitka
uv run python -m nuitka --standalone src/main.py

which spits out an executable.

It's not really the right tool to be including with uv, it is essentially an alternate Python interpreter which compiles Python to C. But it's very easy to add and has worked well for me. And my experience with the developers has been very good. They were quick to add support for Rye when I encountered bugs - https://github.com/Nuitka/Nuitka/issues/2933

my1e5 avatar Aug 07 '24 09:08 my1e5

@my1e5 why do you say that Nuitka would be unsuitable for inclusion with uv?

matterhorn103 avatar Aug 21 '24 07:08 matterhorn103

@matterhorn103, I guess what I mean is that Nuitka is perhaps quite an opinionated way of packaging your Python code into an executable - as it is essentially an alternate Python interpreter which compiles Python to C. Whereas tools like Pyinstaller are more like a bundler - taking the Python interpreter, Python files and dependencies and packaging them into an executable. This is definitely more straightforward than the Nuitka approach, but it does make the Pyinstaller executable very easy to un-package and see the underlying source code (see https://github.com/extremecoders-re/pyinstxtractor). Which might be a caveat needed when including certain 'executable creators' within uv - you need to make users aware that their exe can easily be un-packaged.

In terms of licensing, Nuitka is MIT licensed. But it does also have a commercial tier - which may or may not complicate matters if Astral were to want to bundle it with uv, I don't know.

my1e5 avatar Aug 27 '24 09:08 my1e5

I think this is a duplicate of https://github.com/astral-sh/uv/issues/2799, though there's more discussion here.

zanieb avatar Sep 03 '24 23:09 zanieb

Also want to call out pex, a tool for generating .pex files (Python EXecutable) which are self-contained zipped executable Python environments containing sources, requirements, and dependencies.

https://github.com/pex-tool/pex

martimlobao avatar Sep 04 '24 09:09 martimlobao

I am interested in a solution like this. I think the bundler should be somewhat coupled with uv run since you want to turn any entry point into an executable.

Example

$ uv init --script foo.py
$ uv add requests --script foo.py
$ uv run foo.py
Hello from foo.py!

Then you could make an executable which is composed of

  1. bootstrap code to download uv if needed and run it
  2. target code
  3. uv executable (optional)
  4. python executable (optional from uv python)
  5. populated venv (optional)
$ uv bundle foo.py -o foo --include-uv --include-python --include-dependencies
$ ./foo 
Hello from foo.py!

This above would be a self contained binary (roughly 30MB as uv is ~12MB and python ~17MB).

If downloading stuff on the fly is ok, the bootstrap code would just download uv, then uv would download python, install dependencies and run all during 1st execution.

It should also be possible to create binaries cross platform since uv supports install/sync with --python-platform

$ uv bundle foo.py -o foo.exe --python-platform windows

In a 2nd iteration, it would be nice to be have installers such as to create .dmg (mac) or an .exe windows installer file.

jbvsmo avatar Oct 05 '24 21:10 jbvsmo

what if the command was uv run ruff --output-file ruff.uvx with the -o/--output agument as an indicator to create the binary file rather than execute this tool right now. and when an output file is specified, then include python by default

dbrtly avatar Oct 18 '24 23:10 dbrtly

I love this idea and @jbvsmo proposal for CLI usage.

I am a bit confused on the best choice for the bundling "backend" though, I see different options on the table and I'm not sure, how would you choose one over the other? Should maybe uv propose some protocol for each library to support bundling via uv?

This feature would be great also for bundling pyspark apps (thinking about pex support mentioned by @martimlobao here)

stefanondisponibile avatar Oct 20 '24 17:10 stefanondisponibile

I think the bootstrap option mentioned by @jbvsmo is almost achievable with the existing uv options.

You could do the following:

Within your project folder have a script (written in the scrippting language supported by your operating system) that does the following:

  1. Unmanaged installation of uv (https://docs.astral.sh/uv/configuration/installer/#unmanaged-installations) can be used to just get the uv executable in a subdir of the project folder.
  2. Setup the cache dir to be inside your project folder (https://docs.astral.sh/uv/concepts/cache/#cache-directory)
  3. Then install Python
  4. Install project deps
  5. Use uv run to execute a script within the project enviroment. The easiest option would be to just "run" the package using a __main__.py within the project

Step 1 could of course be skipped, if the uv executable is already recognized within the specified subdir. And step 3-5 are basically what happens when you run uvx. So maybe this is actually a 2 line script.

The great thing is that this script would be project independent! So someone just would need to write a script for each operating system and you could throw these scripts in your project folder.

Then you just need to distribute this project folder and have people execute the script corresponding to their operating system.

This process could then of course be wrapped further by an installer, but seems relatively straight forward to at least address the cases of "Please provide an exe".

PS: If instead of creating a shell script to run the steps above, you use the darkmagic of cosmopolitan you could even create a single binary that dropped into a python package project anywhere makes it run by "double-clicking" the file.

AKuederle avatar Nov 28 '24 13:11 AKuederle

For me as a short term solution it would be enough to have uv run with some options.

For example I have done a program (as module) using PySide6. I can do uv run -m my_module from the folder above the module and it works. Fast, efficient, impressive. But there are two things that can help as an intermediate solution before a uv bundle.

  • Read a compressed file for the full project: uv run --source my_file.zip
  • Allow an option to hide or close the terminal for qt programs: uv run --source my_file.zip --no-terminal

With this solution I could distribute on Windows the files with "Inno Setup", adding uv.exe, the zip file, and maybe a ".bat" file with a logo to execute the uv -run command.

jromal avatar Dec 04 '24 11:12 jromal

As much as I like Pyinstaller (to me it's best of current options), one other thing that it suffers from is that anti-virus vendors are always flagging generic Pyinstaller binaries as malware. This is (at least in part, I think) because Pyinstaller uses the same bootloader binary for all built applications, thus if someone bundles malware with Pyinstaller version X, all other users who bundle legitimate applications using Pyinstaller X can get flagged as malware too. I've seen this hit AV software across both Windows and Linux over the years. Not sure I have any ideas on how Astral could improve that issue, but wanted to bring it up as a current (very annoying) real world challenge.

johnthagen avatar Dec 05 '24 03:12 johnthagen

It's not exactly the same thing, but I've suggested a relatively simple feature to uv that would solve at least a subset of the requirements of delivering a single downloadable executable for a Python program: #10465

Basically it would let uv serve as a limited runtime-only alternative to PyApp.

ceejatec avatar Jan 10 '25 10:01 ceejatec

Thanks for your request and for commenting here.

For me the final tool would be a third party tool, part of uv (for Windows, uv.exe, uvx.exe and the new one, let's call it uvd.exe).

And that third tool should be PyOxidizer. I do not know if you know it. But the single python binaries that Astral uses are a "byproduct" of PyOxidizer. A project from Gregory Szorc (indygreg on GitHub). It is a rust tool for byilding executables including a full python run time environment. Gregory had no time for any of the projects. The python binaries are now generated by Astral. PyOxidizer is not maintained since many months. I considered it as a future better alternative to Pyinstaller or Nuitka or ...

Maybe we could get a simple tool and a complex one with all the hooks for the different packages. Maybe could also have different license models to allow Astral to capitalize the investment. But should be continued as a natural evolution of uv.

But as a you say a wrapper of uvx would do it.

Let's wait and see.

jromal avatar Jan 10 '25 12:01 jromal

Unfortunately https://github.com/indygreg/PyOxidizer only support Python up to 3.10 as specified in the latest stable documentation, except that it looks promising. Latest commit is two months old but you say it is not maintained again?

Nardol avatar Jan 10 '25 13:01 Nardol

I would also be interested in more direct support for using uv as an "installer" for python CLI tools.

I've been using uv in novel ways for this purpose already. I recently wrote a bit about these approaches. Leaving the link here in case folks find it interesting or helpful:

https://aider.chat/2025/01/15/uv.html

paul-gauthier avatar Jan 17 '25 20:01 paul-gauthier

It will be useful if uv add compile command like in bun.js which will produce self contained executable. The only important thing is NOT to work like pyinstaller and extract the contents but somehow load the interpreter into the memory, and tell cpython to run the bundled script directly from the memory. Just like bun.js / nodejs does.

thewh1teagle avatar Jan 27 '25 02:01 thewh1teagle

It's worth noting that it looks like PyOxidizer has been taken over ty Astral-sh - in the form of python-build-standalone.

So, this issue can likely be closed.

davidlanouette avatar Feb 10 '25 20:02 davidlanouette

No, we don't own PyOxidizer — we're just maintaining the standalone Python distributions. These ideas are separate.

zanieb avatar Feb 10 '25 20:02 zanieb

Interesting. That's not how I interpreted the blog post

TL;DR: On 2024-12-17, we'll be taking stewardship of python-build-standalone, Gregory Szorc's foundational project for building and installing portable Python distributions. You can read Gregory's own announcement here.

But, it does explain why the releases seem to only contain python distributions.

davidlanouette avatar Feb 10 '25 20:02 davidlanouette

Yes we are the owners of python-build-standalone, but not PyOxidizer — that's built on top of the standalone distributions.

zanieb avatar Feb 10 '25 21:02 zanieb

This is a bit off topic, but I had a related discussion on discord about this topic and thought I would share A solution to MY problem that could be relevant to others. I feel that python-build-standalone has huge potential QoL improvements for the python package management ecosystem because it (finally) seems to attempt to address the local vs. global installation "problem".

My problem statement: create a zipped folder with standalone python, standalone packages and their dependencies that I can distribute to other users ("download, unzip and run"). Non-goals: a single executable -- for me a folder containing lots of subdirectories/files is fine but an obvious "entry point" is desirable. Obviously the result will be OS and CPU type specific.

Steps I followed:

  1. Download relevant https://github.com/astral-sh/python-build-standalone/releases and unzip
  2. chdir to python subdir
  3. execute e.g. python.exe -m pip install uv (there is no specific need to use uv by the way)
  4. execute e.g. python.exe -m uv pip install <somepackage>
  5. for my application I created somepackage.bat (one directory "up") containing call python/python.exe -m <somepackage>. Obviously shell scripts would be useful on other platforms, and I am sure there are other solutions as well.
  6. zip the parent folder and distribute to others with same OS and CPU type

The reason I like this solution is multifold (1) uses standard package installation methods (2) everything (?) is local to the folder (3) no e.g. pyinstaller-hooks-contrib to maintain. Are there downsides to this approach? Probably, but in my quick testing did not run across any issues -- it "just worked". This is contrast to my prior experiences with nuitka and pyinstaller which work perfectly for projects with pure python dependencies, but can require more "handholding" when that is not the case.

Finally back to the topic of this post -- this would be very nice if supported by uv directly in some form.

p.s. thanks to @geofft for their help

jdegenstein avatar Feb 14 '25 23:02 jdegenstein

The creator of the standalone python (Gregor Szorc) did it for his tool PyOxidizer. A rust substitute of pyinstaller or Nuitka.

I am repeating when I can that Astral should take it and finish it. Gregor (indygreg) has no time and his work was really good. Just unfinished. But he did one of the biggest changes in python. The standalone python. Do not let his work disappear.

jromal avatar Feb 15 '25 08:02 jromal

@jdegenstein it sounds like you want to do uv tool install my_app but have it work even if uv is not installed on the client machine.

dbrtly avatar Feb 15 '25 08:02 dbrtly

(@jromal -- incase you haven't seen, we did assume stewardship of python-build-standalone: https://github.com/astral-sh/python-build-standalone)

charliermarsh avatar Feb 15 '25 13:02 charliermarsh

Since I originally created this issue it's been very popular, and it's also seen many comments offering suggestions.

Naturally it is always a bit presumptuous to attempt to speak on behalf of a diverse and anonymous group, so maybe I'll be contradicted here, but I reckon that the vast majority of those who have voted for this issue or chimed in are united in hoping for something defined not by its technical implementation but by key behaviour – something that is:

  • a single executable (or appears to be)
  • easy-to-use by non-technical users i.e. it it looks and behaves like an application generally does
  • easily distributed through sharing the executable itself i.e. it is fairly self-contained and portable (maybe cross-OS, maybe not)
  • not less performant than running the original Python code with a normal interpreter

This is essentially about being able to take a Python program on our machines and give it to non-programmers, right? It could be achieved with a variety of technical solutions, as shown by the slew of existing tools, and by people's contributions in this thread.

I kind of want to clarify that while many people have been advocating for a wide variety of things, this issue as originally written wasn't advocating for a particular technical solution or requesting anything beyond that listed above.

Going by past form, no doubt any uv implementation would be an excellent one.

(In the original post I suggested that maybe it would use pyinstaller in the background, but that was mostly because it seemed to me a bit rude to ask the team to write a whole new thing from scratch at a time when they were still getting core functionality up and running.)

matterhorn103 avatar Feb 15 '25 16:02 matterhorn103

@matterhorn103 FWIW, this is exactly my use case. I'm building a an "app" for non-programmers. I use uv with hatch to build the distributions. I have a custom hatch build script that kicks off pyinstaller to build the "executable". In my case, I'm using the "one_dir" option because along with the exectuable I have to ship various data files that are needed. One thing I ran into is that running pyinstaller standalone from the command line worked great. My custom build script that issued the same command line arguments was failing to find all the dependencies because of uv's build isolation. I had to set

[tool.uv]
no-build-isolation = true

and then it all worked well. Here's the snippets of relevant code.

hatch_build.py

import subprocess

from hatchling.builders.hooks.plugin.interface import BuildHookInterface


class CustomHook(BuildHookInterface):
    """USed to kick off Pyinstaller as part of a standard uv build.
    """
    def initialize(self, version, build_data):
        cmd_options = ['pyinstaller',
                       '--noconfirm',
                       # add your command line options here
                       'blah.py'   # your script you are wrapping.
                       ]

        subprocess.run(cmd_options, check=False)

pyproject.toml

[tool.hatch.build.targets.sdist.hooks.custom]
    dependencies = ["pyinstaller"]

This should get you what want I think.

tjnd89 avatar Feb 15 '25 17:02 tjnd89