[feature] implement lipo deployer (#52)

[gmeeker]

trafficstars

This is a different approach to #52 where the user deploys once per architecture. The previous attempt used a custom command that handled multiple deployments.

[TheSin-]

I'm trying to test this out with conan 2.0.9. I install them and see it installing

...
Copying file tools_lipo.py to /Users/thesin/.conan2/extensions/deployers/lipo
Copying file lipo.py to /Users/thesin/.conan2/extensions/deployers/lipo
Copying file lipo_add.py to /Users/thesin/.conan2/extensions/deployers/lipo
...

but I do not see an option for install --deploy in install -h and when I try to use it, I get an error about --deploy

conan install . --deploy=lipo . -s arch=armv8
...
conan install: error: ambiguous option: --deploy=lipo could match --deployer, --deployer-folder

I'm still trying to learn conan so sorry if this is a noob thing

Should it be

conan install . --deployer=lipo -s arch=armv8

[TheSin-]

perfect thank you! I'll continue testing now ;)

[gmeeker]

It wasn't ambiguous in 2.0.6 but I've updated the README. If you have any feedback please let me know. I'm probably going to close the earlier custom command approach, but I've been waiting to see if anyone has thoughts about the two approaches.

[WarGloom]

Can we create universal packages without local deploy? Honestly I don't like idea copying everything to build folder. I'm not sure I understand why you decide use deploy over hooks ?

[WarGloom]

if run lipo deployer with generator CMakeToolchain, after lipo_add it overrides CMAKE_OSX_ARCHITECTURES to last used arch

[gmeeker]

The history in #59 is quite long now, and probably not worth reading in its entirety. Most newcomers to Conan seem have the same initial thought: why can't I just set CMAKE_OSX_ARCHITECTURES and get universal binaries? Unfortunately there are thousands of recipes in CCI and many of them use other build systems (see OpenSSL). Some of these can be built by overriding CFLAGS with "-arch x86_64 -arch arm64" but the most reliable method is to simply build each architecture and run lipo on the resulting binaries. I looked at different ways of automating this (without updating thousands of recipes) including hooks (not possible for all recipes) or modifying conan itself (very complicated and invasive).

This deployer is the simplest extension to start testing the lipo code and it still needs additional feedback. I haven't tested iOS at all. I hoped it would be merged by now, but maybe most people are just waiting for a full solution with a conanfile building a universal app?

Beyond this initial test as a deployer, I intend to investigate generating a universal CMake project as you mentioned above (which was possible in 1.x with toolchains but no longer seems to be officially supported). Then connecting the CMake generator to the universal dependencies.

I had a working 1.x solution here (for the universal dependency + CMake toolchain for final build) but it's very kludgy and uses Conan internals. I'd like to do something similar with the public API and new 2.0 features that is clean enough to be accepted to conan-extensions. https://github.com/gmeeker/BuildConan

[MartinDelille]

Hi @gmeeker !

I am very interested in testing your work as soon as my project will be conan 2 ready (my application is still using Rosetta).

[WarGloom]

Ok, Great I've been able to build universal libraries and project except I have to use generator CMakeDeps with CMakeToolchain it defines only the last arch used with lipo_add in cmake files

Great work, thank you, and please add to Readme how to proper install this extension. It's not obvious for newbie

[garyo]

I need to build universal binaries for my conan dependencies on Mac; this looks like it would be useful. Looks it hasn't been merged into main yet; any update on that?

[memsharded]

Hi all,

Sorry, this hasn't been reviewed yet, there are too many things ongoing and the backlog is just too large, we have been trying to prioritize things like the migration of ConanCenter to Conan 2.0.

I think we need to resume this, I'll try to bring this to the team attention hopefully in the next weeks. Thanks for your patience.

[CLAassistant]

All committers have signed the CLA.

[gmeeker]

It sounds like the deployer here is preferred over the custom command in #53 (the implementation details over argument parsing and loading conanfiles were never resolved cleanly.)

Also the iOS support here still needs to be tested. Support for xcarchive and xcframework might be necessary?

[MartinDelille]

I'm now using conan 2.0 and I'm trying to make an univeral binary of my Qt MacOS application. If I understand well, this extension could help me, right ?

[gmeeker]

This deployer can only produce universal binaries for your dependencies, such as Qt. You'll still need to make an Xcode project by hand (or CMake without Conan) and link the deployed libraries. That's probably fine for an iOS only project, but I think most people expect to define their application with a final conanfile too. I haven't looked into that much, but it doesn't appear that Conan's CMake generator allows overriding the architectures (let alone the link directories). I do plan to work on that but it's probably going to be a custom generator that involves some ugly hacks.

[czoido]

Hello @gmeeker,

Thank you very much for your contribution. Although this way of implementing seems valid, I think it would be better to do it through a custom command, as that way we could have an extension with more flexibility to adapt to future user requirements. For example, the idea that you can create universal binaries only of certain libraries from a full deployment, being able to create another command to get information of the binaries like 'lipo info' etc.

I have made a PR taking some fragments of your code as the first draft of an alternative proposal that would work as follows:

# we do the full_deploy of the architectures for which we are going to want to create the universal binary
conan install --requires=mylibrary/1.0 --deployer=full_deploy -s arch=armv8
conan install --requires=mylibrary/1.0 --deployer=full_deploy -s arch=x86_64

# we scan the folder structure and from all the architectures found we create a new
# folder with the resulting structure under the subdirectory ./universal
conan bin:lipo create full_deploy/host --output-folder=universal

Please, take a look at the PR and let us know what you think, it's just the basis from which to incorporate ideas.

In addition to this, the team has talked about starting an investigation with the intention of exploring whether we can provide native support in Conan for universal binaries, although this may take time and does not guarantee that we will add the feature, but we will try to make the effort of the investigation.

Thank you very much, I look forward to your comments!

[gmeeker]

Hmm, I actually started by looking at custom commands but there were technical issues that made it impossible in one go without a deploy. See here: https://github.com/conan-io/conan-extensions/issues/52

The lipo deployer here runs lipo multiple times, so your approach seems more consistent with the way Xcode produces universal binaries.