conan
conan copied to clipboard
conan-io
[feature] PyInstall --onefile support
I would like a flag on pyinstaller.py to toggle the use of --onefile when running the pyinstaller command. This would bundle all components into one executable, a portable monolithic Conan.
I'm no python expert, but from my examination there are a minimal set of changes to the script that would be necessary:
- A command line flag
a. I would nominate
--onefile-conan-client. b. Could be done withargparseif that is not too heavy for this script. - Use the flag to decide to append
--onefile. - Use the flag to alter the path in
_run_bin(). - Make some changes in the packaging step to not look for conan_server.
Im starting here with an issue rather than just doing a PR myself to solicit input on the idea, and to determine if there are other changes that need to be made elsewhere in Conan to get it to behave correctly as a onefile exe. My quick manual testing didn't find any issues with help or install, but I would love some additional insight.
For context, I'm requesting this to simplify the process of building a monolithic Conan executable. Our use case for Conan precludes us from installing Conan on every dev machine and builder. Instead we would like to be able to download every dependency (including Conan) in the CMake configure step. CMake would fetch the monolithic Conan from our artifactory (or official website if available), store it in the project's working directory, set CONAN_USER_HOME also to some location in the working directory, and execute conan install, all driven by CMake.
I have figured out how to manually edit the pyinstall script locally to get these monolithic executables built on windows and Linux, but for the purposes of automation, it would be nice to have this option in the official repo, so its just a matter of cloning the repo, installing requirements via pip, and then running the pyinstall script, Rather than having to edit the script manually.
- [x] I've read the CONTRIBUTING guide.
Hi @TaylorEllington
The first part is doable, and it doesn't have big risk, as it would be an opt-in. Feel free to do a PR if you want, though it is true that we are strongly trying to direct all new features to develop2 branch only, to Conan 1.X only necessary backports, bugfixes and migration helpers will be added.
The second part, I would advice to reconsider. All the new CMake integrations in Conan that will survive in Conan 2.0, i.e CMakeToolchain, CMakeDeps, and the generation of CMakePresets.json is focused on achieving a transparent integration with CMake, strongly pushed by users, in which it is no longer necessary to modify anything in CMakeLists.txt to provide Conan support. This has been achieved with the generation of conan_toolchain.cmake and generation of CMakePresets.json, and the integration seems to be working greatly, with lots of flexibility and power. For example, configuring a project to build different variants (different compiler versions for example, and static/shared, etc) is possible with CMakePresets and integration with VSCode for example is super smooth.
But on the other hand, it means that these files to be fed to CMake need to be generated before. So calling Conan from CMake, which is something we have been providing in the https://github.com/conan-io/cmake-conan cmake-conan wrapper, cannot work this way. So Conan is following a model a bit more similar to other languages, in which the typical process is "git clone + npm install/pip install + build/test/run". I guess it might still be possible to run without the generated toolchain and the presets, but that is a lot of automation work that is already implemented, that otherwise will have to be managed by the user CMake code, and at least for me, scripting in CMake is not my favorite thing 😅 . And it is important to remember that such flow only works when using Conan as a "pure consumer". Using that CMakeLists.txt that calls conan from it, cannot work if you want to create your own package for that project, and that needs to be disabled for that case (see the cmake-conan readme). I would definitely drive things from any other scripting language, like having a build.py`` in the root of all my repos that drives things, decoupled from the CMake step that would focus exclusively on building, but not project automation.
All the new CMake integrations in Conan that will survive in Conan 2.0, i.e CMakeToolchain, CMakeDeps, and the generation of CMakePresets.json is focused on achieving a transparent integration with CMake, strongly pushed by users, in which it is no longer necessary to modify anything in CMakeLists.txt to provide Conan support.
To clarify, are you saying that with the advent of Conan 2.0, my intended use paradigm will no longer work? We are just now looking into adopting Conan, but if the next significant version of Conan is going to require the npm model, and it cant be driven entirely by CMake, we may need to look at selecting a different tool. We like the flow of running the wrapper's install function, and then importing conanbuildinfo_multi.cmake. Idealy most members of our team wont need to interact with conan
I guess it might still be possible to run without the generated toolchain and the presets, but that is a lot of automation work that is already implemented, that otherwise will have to be managed by the user CMake code, and at least for me, scripting in CMake is not my favorite thing 😅
CMake is definitely not my favorite language either, but we had a need to reduce the footprint of necessary "build bootstrapping" installations. We looked at features of the various tools we were using, which at the time included python and CMake. We chose to eliminate python, and reworked our entire build to be driven and orchestrated by CMake.
All we need from Conan in the context of this build, is to pull down packages. We are fine maintaining the conanfile, and files with our profiles and feeding that to Conan in CMake, We intend to use the cmake_multi generator. Hopefully these features are not going away with the shift to 2.0.
I would definitely drive things from any other scripting language, like having a build.py
we had a build.py, we decided that we did not want to have both a python dependency and a CMake dependency to build, so we removed it.
To clarify, are you saying that with the advent of Conan 2.0, my intended use paradigm will no longer work? We are just now looking into adopting Conan, but if the next significant version of Conan is going to require the npm model, and it cant be driven entirely by CMake, we may need to look at selecting a different tool. We like the flow of running the wrapper's install function, and then importing conanbuildinfo_multi.cmake.
Yes, cmake_multi generator is also removed in Conan 2.0. Only CMakeDeps (which is multi-config, but it is based on xxxx-config.cmake files and find_package() it no longer works doing include(conanbuildinfo.cmake)) will survive, but that is not an issue, the "dependencies" part of the flow will still work fine.
The part that will be missing is the setting up of the configuration via toolchain and presets. I am not saying that it will not be possible, just that you will probably have to do extra work, because you will not leverage a lot of builtin functionality. What will be just 1 command line command conan install . for many users will probably translate to a bunch of CMakeLists.txt lines.
Idealy most members of our team wont need to interact with conan
We did carefully consider all pros and cons of this decision, because there are many cmake-conan wrapper users that enjoy this flow. For us, it is kind of a flashback of the svn->git migration. I did myself some svn->git migrations for teams, and I have learned from many others colleagues doing the same. It was very usual at the beginning to work with git wrappers that allowed to kind of maintain svn-like flow and commands, hiding git "complexities" from users, because developers were heavily complaining that git was too complicated. At the end of the day, and even if IDEs can also help, it turns out that professional developers should learn the git basics anyway, sooner or later. We strongly feel the same about this, and talking from the experience of growth, with many thousands of companies using Conan, seeing Conan advertised in job postings, learning from talking hundreds of hours with our users... that is better to embrace the fact that a package manager if used, becomes an important part of the developers flow, and hiding it only leads to frustration both for developers and for devops teams supporting those developers. Moreover we see a large amount of users happily embracing automation with Python and getting complexity out of CMake, and apparently the ones that automate with CMake tend to struggle more in the midterm.
Of course, this is only our opinion. Sure you can move forward with that approach, using exclusively the CMakeDeps generator (you don't need to use the CMakeToolchain as it will be useless anyway). And as long as you use Conan only to pull dependencies, the moment you intend to create your own Conan packages, you either need to allow Conan drive the build (Conan calls CMake and use the toolchain), or you will need to use conan export-pkg to package the local binaries you compiled with CMake (but that approach doesn't scale to --build=missing entire dependency graphs, for example). In all cases it is indeed possible, as one of the advantages of Conan is that it is extremely flexible, but it is just that some flows are intended to be more optimized and smooth than others, and might require less work.
We aren't looking to package with Conan, our interest and usage is mostly to replace custom download scripts and to make use of pre-built binary dependencies. So as long as CMake can download a one file executable and run Conan install, then get targets. Conan will work for us.
Out of curiosity, what is the timeline for 2.0? I couldn't find a launch target on the wiki.
Also, when 2.0 launches, will packages still be compatible with 1.x versions? or will there be breaking changes?
Of course, this is only our opinion.
As a data point for any further discussions you all have, I have a few personal pieces of feedback, most of these points come from the perspective of my experience with CMake, C++, and associated tools. I understand you all have your direction and this is unsolicited, so feel free to ignore 😅
As a strong proponent of the "Do one thing and do it well" philosophy, a package manager's responsibilities begin and end with downloading packages, making them accessible, versioning, and maybe packaging for upload. CMake, the way we use it, is responsible for building a project, dependency management is a step in that process. To my mind, flipping the script, and having a package manager drive the build feels like the cart before the horse. You make the comparison of Conan to Git, where the learning process, while difficult, is a necessary part of a Software Engineer/developer's maturity. I feel that the same argument can be applied to CMake. Using Python to do things that CMake can do (albeit with a pretty awful syntax, ill concede that point), is -to me- just as messy as a svn-like wrapper around git behavior.
I won't begrudge others using Conan this way, if it makes life easier for them, as long as Conan can be used the way ive outlined in this thread.
I should have the PR ready late this week, if not early next.
Out of curiosity, what is the timeline for 2.0? I couldn't find a launch target on the wiki.
We will release 2.0-beta.2 shortly, and we expect at least 2-3 more iterations (1 per month) to have the Conan client tool ready. The bottleneck might be the readiness of ConanCenter packages
Also, when 2.0 launches, will packages still be compatible with 1.x versions? or will there be breaking changes?
There is a common syntax in 1.X that will be forward compatible with 2.0. The idea is that the same recipes can work simultaneously with the 2 versions. It is necessary to use a subset of the 1.X features, dropping some legacy tools and use some more modern alternatives, but already available in 1.X.
As a strong proponent of the "Do one thing and do it well" philosophy, a package manager's responsibilities begin and end with downloading packages, making them accessible, versioning, and maybe packaging for upload. CMake, the way we use it, is responsible for building a project, dependency management is a step in that process.
The things is that with 2 requirements, that are pretty strongly required by a very large number of Conan users, this is not true. These requirements are being able to work and integrate with any build system, natively, and being able to build missing binaries from source when necessary. If you depend on a bunch of Conan packages, and for some reason there are not existing pre-compiled binaries, Conan should be able to build them from source (like using --build=missing, and build them for the configuration, architecture, build_type, options, etc). Thus, Conan should be able to launch the build system of a package with the desired configuration, defined by the consumer. But Conan is driving that build, telling Meson, Autotools, Visual Studio: "hey, you should build a binary for architecture xyz, a shared library, in Debug mode, and with compiler UVW, version 1.2.3". Furthermore, when you are installing and building a dependency graph of many packages, each package build system shouldn't individually define and install their dependencies, because that can easily lead to inefficiencies, but worse things like inconsistencies like version conflicts. So dependencies versions cannot be hardcoded in the build scripts, otherwise they will not be able to be overriden or updated by any dependency resolution mechanism external to those build scripts.
Of course if you don't ever plan to use the --build=missing or any kind of other --build=xxx feature, this wouldn't be that relevant, but this is a very important use case for many Conan users.
Many thanks for the interesting discussion!
Reopening this discussion because ive hit another snag. So upon further testing this does not appear to be a valid workflow. a standalone Conan executable is not viable as it when i try to install, i end up with errors like:
Error loading conanfile at '<path_redacted>\.conan\data\expat\2.4.8\_\_\export\conanfile.py': Unable to load conanfile in <path_redacted>\.conan\data\expat\2.4.8\_\_\export\conanfile.py
File "<frozen importlib._bootstrap>", line 702, in _load
[cmake] File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
[cmake] File "<frozen importlib._bootstrap_external>", line 783, in exec_module
[cmake] File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[cmake] File "<path_redacted>\.conan\data\expat\2.4.8\_\_\export\conanfile.py", line 8, in <module>
[cmake] from conan.tools.scm import Version
[cmake] ModuleNotFoundError: No module named 'conan.tools.scm'
so it looks like with out the conan/conans modules themselves that individual conanfiles will blow up. Any advice how to work around this?
the pyinstaller only bundles the exe, im guessing that i would need to also bundle these modules and make them discoverable on the pythonpath?
@TaylorEllington This seems related to missing --hidden-imports in the pyinstaller.py. They have been added recently for minor 1.51.2 patch release, if you want to update and try again.
Hi @TaylorEllington
What is the status of that? Still want to have the --onefile support in pyinstaller.py?
For information, now that 2.0 is finally out, we are resuming work on the CMake integration, and we will shortly be investigating the best way to have the previous Conan 1.X cmake-conan integration, so it is possible to drive Conan from CMake directly too in 2.0.
Sorry for going radio silent. We had a priority change in the team, and the pilot project for overhauling our dependency download was pushed back. The POC PR I was working on in Aug got scrapped, loss of the CMake driven interaction in an upcoming version caused a decrease in interest.
If the CMake interactions are working when this update for our team becomes a priority again, we may take another look at Conan.
Ok, then I suggest to close this as the --onefile wouldn't be necessary now, and I'd suggest tracking news in https://github.com/conan-io/cmake-conan and in our social channels (slack, twitter), this will be announced when there is progress, and also documented in docs.conan.io. And of course, don't hesitate to re-open or create a new ticket for any further question or request that you might have. Many thanks!