App-perlbrew Manage Perl modules with "Universal Binary" (e.g. arm64, x86

Does App::perlbrew support the management of the *.bundle macOS "Universal Binary" (e.g. arm64, x86_64) files in Perl modules?

"StackOverflow: Install & update a Perl module as "universal" (x86_64, arm64)?"
ExtUtils::MakeMaker -- Generate a Makefile with "Universal Binary" (e.g. arm64, x86_64) directives? #445

Apr 26 '23 23:04 systemresearch

perlbrew itself does not do compilation of Perl modules but only tweaks enough env variables to make cpanm or cpan install modules to the right place -- either a "site_perl" directory or a local::lib directory.

Also the complication setup of a module depends on the content of Makerile.PL or Build.PL of the module. To my knowledge, I don't think any comonnly-used, simple, setup would produce universal binaries at the end.

The "ARCHFLAG" approach you've mentioned in https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/issues/445 seems to lead to some viable solution. But currently this variable is not special to perlbrew.

Apr 30 '23 07:04 gugod

I did find that ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' worked OK with cpanm. Also, on macOS, binary Perl .bundle modules can be found and checked with a find /Library/Perl -name "*.bundle" | xargs file approach.

So, while perlbrew does not internally have any specialized code for dealing with universal binaries, the questions to consider might be:

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?
And, if yes, should this particular use case be mentioned in the perlbrew docs?

As FYI, I am not currently a user of perlbrew. I found perlbrew in a broad survey for understanding and possible tools for dealing with the Perl univeral binary issues. Thus, I had posted the issue|questions here. Thank you for your reply.

May 15 '23 00:05 systemresearch

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

I haven't test it myself but I believe so. Since ARCHFLAGS is not specially handled by perlbrew itself, by properly export-ing this variable in the shell, the compilation process triggered by perlbrew install should be able to see this variable and respond correctly.

And, if yes, should this particular use case be mentioned in the perlbrew docs?

Since perlbrew itself is meant for personal use on a single computer instead of producing / distributing binary builds, I see little value of building universal-binaries. But maybe that's something people would want. It would still be nice to mention this approach (once verified) in a document. Eventually we will find it to be useful. :)

May 15 '23 01:05 gugod

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

May 16 '23 12:05 Leont

@gugod Good point on scope, here, there situation...

... personal use on a single computer instead of producing / distributing binary builds ...

For clarification, this issue actually is a "personal use on a single computer" situation. I'm not trying to create any binaries for distribution.

Goal: personal use, single user, standard-release Perl installation without Instruction Set Architecture (ISA) runtime issues... on Rosetta enabled macOS.

After a user has enabled Rosetta 2 on an Apple Silicon computer, there are two Instruction Set Architectures (ISA) which automatically dispatch without requiring any additional user interaction.

A Rosetta 2 enabled Apple Silicon computer has automatic dispatch on either the native arm64 (arm64e) ISA or the rosetta x86_64 ISA from the mach-o *.bundle binary. Any "Universal" mach-o *.bundle binaries which include arm64/arm64e/x86_64 will avoid ISA runtime issues for the general (technical and non-technical) user.

At a technical level, which binaries slices (arm64|arm64e|x86_64) are in each *.bundle can be found from the macOS terminal command line:

BREW_INSTALL_DIR = /Library/Perl
find $BREW_INSTALL_DIR -name "*.bundle" | xargs file
# -- or, check the whole system --
find / -name "*.bundle" 2> /dev/null | xargs file

May 16 '23 23:05 systemresearch

@Leont A shapshot of findings relative to the current state of "universal binaries" and perl...

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Good question… there appears to be a least three approaches to create mach-o universal binary loadable library bundles in the Perl eco-system:

ARCHFLAGS
lipo - universal binary tool
ccflags/ldflags/lddlflags

ARCHFLAGS

Use of ARCHFLAGS was (re-)discovered in the current macOS 13.3 Ventura (but long-ago written) perlmacosx man page.

man-perlmacosx-64bit

The above information is clearly not recent because it speaks to PowerPC/Intel and 32/64 bit support. Even so, the above information was found to still successfully work for Perl as follows:

env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perl Makefile.PL

make
### An Apple Silicon machine can test all three architectures:
arch -x86_64 make test
arch -arm64  make test
arch -arm64e make test
### An Apple Intel machine can only test its native architecture:
make test

(sudo) make install

An addition, I later found that ARCHFLAGS worked with a cpanm install of Finance::Quote for GnuCash.

(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpan App::cpanminus
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Test2
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm Finance::Quote
(sudo) env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' cpanm JSON::Parse

### verify universal binaries
find /Library/Perl -name "*.bundle" | xargs file

Side note: It was @Leont comment "If you can make -arch x86_64 -arch arm64 work…" that lead to my grepping terms like arch x86_64 which found the forgotten & obscure perlmacosx man page.

lipo - universal binary tool

The current Apple Developer documentation for "Building a Universal macOS Binary" provides information to use the lipo tool in the makefile:

x86_app: main.c
    $(CC) main.c -o x86_app -target x86_64-apple-macos10.12
arm_app: main.c
    $(CC) main.c -o arm_app -target arm64-apple-macos11
universal_app: x86_app arm_app
    lipo -create -output universal_app x86_app arm_app

Note: You can build a universal binary on either an Apple silicon or Intel-based Mac computer, but you cannot debug the arm64 slice of your binary on an Intel-based Mac computer. It’s possible to debug both slices of a universal binary on Apple silicon.

man lipo

lipo-man-page

ccflags/ldflags/lddlflags

-- Proof of Concept --

A proof-of-concept ccflags/ldflags/lddlflags example did demonstrate that Perl [arm64, x86_64] univeral binaries could be created. For me, this indicated that Perl universal binaries are possible … that was great news! However, a hand-edit-the-build-sequence approach did not look convenient to use on a general wide scale.

#! /bin/bash
arch_opt="-arch x86_64 -arch arm64"
opt1="-I./Encode -fno-common -DPERL_DARWIN -fno-strict-aliasing"
opt2="-mmacosx-version-min=12.0 -fstack-protector-strong"
opt3="-pipe -DPERL_USE_SAFE_PUTENV -Wno-error=implicit-function-declaration -O3"
opt4=-DVERSION=\"3.19\"
opt5=-DXS_VERSION=\"3.19\"
opt6="-I/opt/homebrew/Cellar/perl/5.34.0/lib/perl5/5.34.0/darwin-thread-multi-2level/CORE"
ccopts="$arch_opt $opt1 $opt2 $opt3 $opt4 $opt5 $opt6"
ldopts="$arch_opt $opt2"
cc -c  $ccopts Encode.c
cc -c $ccopts def_t.c
cc -c $ccopts encengine.c
cc -bundle -undefined dynamic_lookup $ldopts \
   Encode.o def_t.o encengine.o -o blib/arch/auto/Encode/Encode.bundle
file blib/arch/auto/Encode/Encode.bundle

-- ExtUtils::FakeConfig Config_u --

The next finding was ExtUtils::FakeConfig Config_u.pm which also used ccflags and lddlflags:

package Config_u;
 
require ExtUtils::FakeConfig;
require Config;
 
my %values =
  ( lddlflags => ' -arch i386 -arch ppc ' . $Config::Config{lddlflags},
    ccflags   => ' -arch i386 -arch ppc ' . $Config::Config{ccflags},
    );
 
ExtUtils::FakeConfig->import( %values );

The MConfig_u use synopsis looked simple enough:

perl -MConfig_u Makefile.PL
make
make test
make install

However, MConfig_U was only maintained from 2002 to 2008 .AND. had a caveat that lipo would be the "safest" approach:

Note that the safest way to build Universal binaries is to compile the modules separately and then use lipo(1) to merge the resulting .bundle files.

Note: This was the point in time where I inquired if ExtUtils::MakeMaker could "Generate a Makefile with "Universal Binary" (e.g. arm64, x86_64) directives?". It seemed like the place where Makefile.pl transitions to Makefile would be generally helpful to the larger (macOS) Perl ecosystem.

-- Perl Variables --

It was news to me that ccflags/ldflags/lddlflags might be built into perl itself...

What does perl -V:ccflags -V:ldflags -V:lddlflags return?

Ideally, you would put them in each of those variables when building perl, then you don't have to worry about it later.

Interesting. Yet, puzzles remain:

Could this really be a easy as adding the flags to perl for the general community?
- Or, does Apple know some reason to only add the -bundle flag to perl?
General releases of Apple's system perl and Homebrew perl had only a -bundle flag to indicate creation of a mach-o bundle. Otherwise, no universal binary flags are present.
Creating and maintaining one's own build of perl seems like a significant undertaking.

Summary

ARCHFLAGS. I'm currently using the ARCHFLAGS approach because it works at a good-enough level for an end user of software which uses perl. Successful work-around status.
lipo. A makefile with lipo is the stated (preferred?) approach in Apple developer documents. However, this approach has not been found to be in use for perl.
ccflags/ldflags/lddlflags. This would be OK if the flag details became part of general perl release and worked with the downstream Makefile.PL/Makefile builds. … however, it's unclear how to get the right persons' attention on this option.

May 17 '23 00:05 systemresearch

@Leont Switching from a user to implementer perspective...

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Perhaps a meaningful difference is that ARCHFLAGS have broader designation within a given Makefile for flow & logic than ccflags/ldflags/lddlflags? i.e. ccflags/ldflags/lddlflags have a more narrow designation to the flags of a specific build tool.

Some GitHub related code searches:

May 17 '23 17:05 systemresearch

Is perlbrew compatible with an env ARCHFLAGS='-arch arm64 -arch arm64e -arch x86_64' perlbrew … approach?

Why use ARCHFLAGS and not put it in ccflags/ldflags/lddlflags?

Uniformity is a good thing. A uniform way of building perl and building perl modules is ideal.

For example, if I wanted to create a macOS distribution of perl 5.32.1 and then install modules that are also universal and then distribute that perl so that both x86_64 and arm64e customers could use it, then I'd want to use the same mechanism to perform all operations so that universal binaries were produced. I think that's the first thing people would think of: one way to identify my intentions for all compilations.

Aug 03 '23 17:08 jlholt

Manage Perl modules with "Universal Binary" (e.g. arm64, x86_64)