Remove bloat from our binaries

[samuelpmishLLNL]

re: https://github.com/LLNL/serac/issues/761

It looks like by default, uberenv builds everything as RelWithDebInfo, so the TPLs produce huge binaries:

sam@provolone:~/code/serac_libs_before/gcc-9.3.0/conduit-0.8.3-wqhg2i6ady424csssfpjlggcjflt42jx/lib$ ls -la
...
-rw------- 1 sam sam  64230114 Aug  4 13:00 libconduit.a
-rw------- 1 sam sam 185510866 Aug  4 13:00 libconduit_blueprint.a
-rw------- 1 sam sam  41864046 Aug  4 13:01 libconduit_blueprint_mpi.a
-rw------- 1 sam sam  47961410 Aug  4 13:01 libconduit_relay.a
-rw------- 1 sam sam   4944286 Aug  4 13:00 libconduit_relay_mpi.a
-rw------- 1 sam sam  27381396 Aug  4 13:01 libconduit_relay_mpi_io.a

vs.

sam@provolone:~/code/serac_libs_after/gcc-9.3.0/conduit-0.8.3-6ilpkq7rsb23lavuks2wuv45wppjkbv4/lib$ ls -la
...
-rw------- 1 sam sam 3797592 Aug 11 12:44 libconduit.a
-rw------- 1 sam sam 5196542 Aug 11 12:45 libconduit_blueprint.a
-rw------- 1 sam sam 2186540 Aug 11 12:45 libconduit_blueprint_mpi.a
-rw------- 1 sam sam 2122044 Aug 11 12:45 libconduit_relay.a
-rw------- 1 sam sam  318550 Aug 11 12:44 libconduit_relay_mpi.a
-rw------- 1 sam sam 1543462 Aug 11 12:45 libconduit_relay_mpi_io.a

so about a ~20x reduction in TPL binary size. Linking against these also was increasing the size of our unit tests by more than 100MB+, each.

Before: (Release)

-rwxrwxr-x  1 sam sam 140083584 Aug 10 17:08 serac_solid_functional_boundary
-rwxrwxr-x  1 sam sam 140348240 Aug 10 17:08 serac_solid_functional_finite_diff
-rwxrwxr-x  1 sam sam 141273528 Aug 10 17:08 serac_solid_reuse_mesh
-rwxrwxr-x  1 sam sam 141249672 Aug 10 17:08 serac_solid_sensitivity
-rwxrwxr-x  1 sam sam 141744216 Aug 10 17:08 serac_thermal_functional
-rwxrwxr-x  1 sam sam 140299368 Aug 10 17:08 serac_thermal_functional_finite_diff
-rwxrwxr-x  1 sam sam 141239552 Aug 10 15:48 serac_thermal_mechanics_functional
-rwxrwxr-x  1 sam sam 141422328 Aug 10 17:08 serac_thermal_solid_solver
-rwxrwxr-x  1 sam sam 141432040 Aug 10 17:08 serac_thermal_solver

After: (Release)

-rwxrwxr-x  1 sam sam  24533632 Aug 11 13:28 serac_solid_functional_boundary
-rwxrwxr-x  1 sam sam  24793928 Aug 11 13:28 serac_solid_functional_finite_diff
-rwxrwxr-x  1 sam sam  25588424 Aug 11 13:28 serac_solid_reuse_mesh
-rwxrwxr-x  1 sam sam  25551344 Aug 11 13:28 serac_solid_sensitivity
-rwxrwxr-x  1 sam sam  26167368 Aug 11 13:29 serac_thermal_functional
-rwxrwxr-x  1 sam sam  24750488 Aug 11 13:28 serac_thermal_functional_finite_diff
-rwxrwxr-x  1 sam sam  24462568 Aug 11 13:28 serac_thermal_solid_functional
-rwxrwxr-x  1 sam sam  25733136 Aug 11 13:28 serac_thermal_solid_solver
-rwxrwxr-x  1 sam sam  25746976 Aug 11 13:28 serac_thermal_solver

I tried passing "build_type=Release" as part of the spack spec, but it seemed to do nothing.

This is probably the wrong way to rebuild the TPLs without debug info, but it's an attempt to get the ball rolling.

[samuelpmishLLNL]

I forgot to mention, this change seems to upset the codevelop build. When configuring cmake with these Release-build TPLs, I get:

...
[cmake] -- Looking for NetCDF ...
[cmake] --    in NETCDF_DIR = /home/sam/code/serac_libs_after/gcc-9.3.0/netcdf-c-4.7.4-q6ah5b4fzzqrlp2wscviug2nh5av3zc6
[cmake] -- NetCDF: looking for required package: HDF5, COMPONENTS: C;HL
[cmake] CMake Error at mfem/config/cmake/modules/MfemCmakeUtilities.cmake:540 (message):
[cmake]    *** HDF5: configuration RELWITHDEBINFO not found. Set HDF5_IMPORT_CONFIG from the list: RELEASE.
...

[codecov-commenter]

Codecov Report

Merging #764 (a147ed8) into develop (8b66066) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #764   +/-   ##
========================================
  Coverage    94.90%   94.90%           
========================================
  Files          142      142           
  Lines         9151     9151           
========================================
  Hits          8685     8685           
  Misses         466      466           

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jamiebramwell]

Thanks for looking into this!

[samuelpmishLLNL]

This is not ready to be merged, as it still has that problem with the codevelop build (although codevelop seems to be broken for me on develop as well, maybe I'll make an issue for that as well)

[white238]

I forgot to mention, this change seems to upset the codevelop build. When configuring cmake with these Release-build TPLs, I get:

...
[cmake] -- Looking for NetCDF ...
[cmake] --    in NETCDF_DIR = /home/sam/code/serac_libs_after/gcc-9.3.0/netcdf-c-4.7.4-q6ah5b4fzzqrlp2wscviug2nh5av3zc6
[cmake] -- NetCDF: looking for required package: HDF5, COMPONENTS: C;HL
[cmake] CMake Error at mfem/config/cmake/modules/MfemCmakeUtilities.cmake:540 (message):
[cmake]    *** HDF5: configuration RELWITHDEBINFO not found. Set HDF5_IMPORT_CONFIG from the list: RELEASE.
...

This may be fixed in MFEM already... I'll test it out in a bit.

[samuelpmishLLNL]

the TPLs built by this branch seem to be working on CPU builds (without and without "codevelop"), and on GPU builds without "codevelop" (see: https://github.com/LLNL/serac/issues/767).

For reference, the before and after of the spack build directories:

sam@provolone:~/code$ du -h -d 1 serac_libs/builds/ | sort -h
4.0K	serac_libs/builds/spack-src
8.0K	serac_libs/builds/spack-stage-el3oa0_4
236K	serac_libs/builds/spack-stage-ca-certificates-mozilla-2022-03-29-mwkqkggqeilj5bqglgdn5fmf72lmnp2f
1.4M	serac_libs/builds/spack-stage-camp-2022.03.0-7hqhqtkjbt6m2cv52njwiwebbb4nsg3i
2.2M	serac_libs/builds/resource-luarocks-6iqsi2j2ypb64qo4rhevnfjerbjmmgzp
5.0M	serac_libs/builds/spack-stage-lua-5.3.5-6iqsi2j2ypb64qo4rhevnfjerbjmmgzp
8.0M	serac_libs/builds/spack-stage-blt-0.5.1-xcp5gmah5yixf6qo3nive7g5g2y37nbo
17M	serac_libs/builds/spack-stage-raja-2022.03.0-jxqnqqspagvrmbk6gkk2yge4i3awuppy
20M	serac_libs/builds/spack-stage-readline-8.1.2-32ctcvpyazwkbz4xyxkhavglbl3uly3g
25M	serac_libs/builds/spack-stage-metis-5.1.0-4hoi752pxl7gj26ucl65jsnwhr5yser6
36M	serac_libs/builds/spack-stage-pkg-config-0.29.2-rgh7cy46wmiq7tknmn36phnq6xz6b4lr
43M	serac_libs/builds/spack-stage-parmetis-4.0.3-2dej4tl25yfkdvvaujbvbxkbdeevgyhx
69M	serac_libs/builds/spack-stage-hypre-2.18.2-nvshdeep6ekl2q66bhiqjobbgjd4kvfi
88M	serac_libs/builds/spack-stage-ncurses-6.2-wkhghzhgx2sfmjvjrkznnfohgz4ptyro
100M	serac_libs/builds/spack-stage-sundials-5.7.0-y4ae5tlblcigkpo6noxpfm7nbvkawpfb
102M	serac_libs/builds/spack-stage-superlu-dist-6.1.1-22n2wyxgssrhukhy7u54idoxayphxhgg
103M	serac_libs/builds/spack-stage-umpire-2022.03.1-zdjv3zb7yzwfsky6w22zd7igiz4euo6l
118M	serac_libs/builds/spack-stage-netcdf-c-4.7.4-jh7q2gb5spjv2ixgmyfxk7zsijlgg4ly
146M	serac_libs/builds/spack-stage-curl-7.83.0-mtdczg2j43eolqoj6kjghm4uohatvtj2
149M	serac_libs/builds/spack-stage-openssl-1.1.1q-evoqyre24q7tetfoxsxsy7ovylftkrzq
269M	serac_libs/builds/spack-stage-mfem-4.3.0.1-5ymc6zfbt4jj75oqwbq3ozavxf7c7ttf
464M	serac_libs/builds/spack-stage-hdf5-1.8.21-vi67w2sajactgflts5jbfkvjjzs3mzhg
1.1G	serac_libs/builds/spack-stage-conduit-0.8.3-wqhg2i6ady424csssfpjlggcjflt42jx
1.7G	serac_libs/builds/spack-stage-axom-0.6.1.5-7in2a55papqbajwkygesffzggiukvqfu
4.4G	serac_libs/builds/

sam@provolone:~/code$ du -h -d 1 serac_libs_bloatless/builds/ | sort -h
4.0K	serac_libs_bloatless/builds/spack-src
8.0K	serac_libs_bloatless/builds/spack-stage-6fokhr31
236K	serac_libs_bloatless/builds/spack-stage-ca-certificates-mozilla-2022-03-29-mwkqkggqeilj5bqglgdn5fmf72lmnp2f
1.4M	serac_libs_bloatless/builds/spack-stage-camp-2022.03.0-7hqhqtkjbt6m2cv52njwiwebbb4nsg3i
2.2M	serac_libs_bloatless/builds/resource-luarocks-6iqsi2j2ypb64qo4rhevnfjerbjmmgzp
5.0M	serac_libs_bloatless/builds/spack-stage-lua-5.3.5-6iqsi2j2ypb64qo4rhevnfjerbjmmgzp
8.0M	serac_libs_bloatless/builds/spack-stage-blt-0.5.1-xcp5gmah5yixf6qo3nive7g5g2y37nbo
17M	serac_libs_bloatless/builds/spack-stage-raja-2022.03.0-jxqnqqspagvrmbk6gkk2yge4i3awuppy
20M	serac_libs_bloatless/builds/spack-stage-readline-8.1.2-32ctcvpyazwkbz4xyxkhavglbl3uly3g
22M	serac_libs_bloatless/builds/spack-stage-umpire-2022.03.1-4df6ziu2r7jkwrwicfpe3kw7xhsvngrw
25M	serac_libs_bloatless/builds/spack-stage-metis-5.1.0-4hoi752pxl7gj26ucl65jsnwhr5yser6
36M	serac_libs_bloatless/builds/spack-stage-pkg-config-0.29.2-rgh7cy46wmiq7tknmn36phnq6xz6b4lr
43M	serac_libs_bloatless/builds/spack-stage-parmetis-4.0.3-2dej4tl25yfkdvvaujbvbxkbdeevgyhx
43M	serac_libs_bloatless/builds/spack-stage-superlu-dist-6.1.1-ywrfhbpbx7cgbbohv26ywmwnwt3lko6h
68M	serac_libs_bloatless/builds/spack-stage-netcdf-c-4.7.4-o6y7fev67kitdnjsohnodlcwdweayumi
69M	serac_libs_bloatless/builds/spack-stage-hypre-2.18.2-nvshdeep6ekl2q66bhiqjobbgjd4kvfi
77M	serac_libs_bloatless/builds/spack-stage-sundials-5.7.0-udhnsxghkt5bpgbex7zwy7f3jfsj7j4z
88M	serac_libs_bloatless/builds/spack-stage-ncurses-6.2-wkhghzhgx2sfmjvjrkznnfohgz4ptyro
116M	serac_libs_bloatless/builds/spack-stage-conduit-0.8.3-7qdt2js27nq7gdnr26hl5e2nl2e4jzat
147M	serac_libs_bloatless/builds/spack-stage-curl-7.83.0-mtdczg2j43eolqoj6kjghm4uohatvtj2
149M	serac_libs_bloatless/builds/spack-stage-openssl-1.1.1q-evoqyre24q7tetfoxsxsy7ovylftkrzq
200M	serac_libs_bloatless/builds/spack-stage-hdf5-1.8.21-dwwbzovel6phzd7et25t2g226dl6ef2w
272M	serac_libs_bloatless/builds/spack-stage-mfem-4.3.0.2-waqhtt7kfsgqqfeexifbonyc6yvlpr6v
417M	serac_libs_bloatless/builds/spack-stage-axom-0.6.1.6-rfmmsr6jb447jnjbjkqu32xpezvyx237
1.8G	serac_libs_bloatless/builds/

so, a significant reduction, especially in conduit (1.1G -> 0.1G).

[samuelpmishLLNL]

Looking at the remaining sources of large files:

417M serac_libs_bloatless/builds/spack-stage-axom-0.6.1.6-rfmmsr6jb447jnjbjkqu32xpezvyx237

• 200MB of data files we don't use (this can be solved by CMake's ExternalData module)
• 100MB of git history
• 40MB of unused executables in axom/spack-build/bin (can we stop these from being built?)

272M serac_libs_bloatless/builds/spack-stage-mfem-4.3.0.2-waqhtt7kfsgqqfeexifbonyc6yvlpr6v

• 140MB of git history
• a lot of object files that are hard to separate due to the in-source build. I wish mfem would use their CMake build system for the spack package so this wouldn't be an issue.

200M serac_libs_bloatless/builds/spack-stage-hdf5-1.8.21-dwwbzovel6phzd7et25t2g226dl6ef2w As far as I can tell, we only need ~ 7MB of static libraries and some headers from HDF5. There's about an extra 60MB+ of unused binary utilities in spack-build-***/bin, and 50 MB of unused test data in spack-src/tools. Ideally, we wouldn't be building the tests and utilities, since we don't use them.

147M serac_libs_bloatless/builds/spack-stage-curl-7.83.0-mtdczg2j43eolqoj6kjghm4uohatvtj2 149M serac_libs_bloatless/builds/spack-stage-openssl-1.1.1q-evoqyre24q7tetfoxsxsy7ovylftkrzq We shouldn't even be building these libraries, as they are readily available in package managers:

sam@provolone:~/$ which curl
/usr/bin/curl

---

The takeaway here is:

• shallow git clones or zipped sources could save several hundred MB in downloads and storage per TPL build.
• we're building quite a few tests, utilities, and executables that we don't use (axom and HDF5 are most relevant)
• we don't use axom's large data files, it would be nice to avoid downloading and storing these

[white238]

Now that I am slightly more awake...

I am not sure what would happen with the current solution if we turn on our debug variant. I think we would end up with something like this axom build_type=Release build_type=Debug. Without testing it I'm not sure Spack will like that and it is also just weird.

Basically they need to be added and with the when parameter:

depends_on("axom build_type=Release", when="~debug")
depends_on("axom build_type=Debug", when="+debug")

Do you mind if I push a fix to your branch @samuelpmishLLNL ?

[samuelpmishLLNL]

I am not sure what would happen with the current solution if we turn on our debug variant.

Good question, I didn't build the debug variant. If "codevelop" is working, I think that's preferable to having an entire separate TPL build (since we don't really want to debug all the TPLs, just axom+mfem and maybe hypre).

Do you mind if I push a fix to your branch @samuelpmishLLNL ?

Not at all. Like I said earlier, this PR is probably a bad implementation, given that I don't really understand spack packages.

[white238]

I went a little crazy with the variant forwarding but this significantly cleans up our dependencies.

Pre-change:

380K    ./adiak-0.2.1-7mkgrnxn2ptgu2krd4tzravn6nmtmpal
443M    ./axom-0.6.1.6-5o4srwqxmwrhewzabzux2tyvxtp32ggw
7.3M    ./blt-0.5.1-7sde54yl6bh46taa4xfujpkiodmxymb5
276K    ./ca-certificates-mozilla-2022-03-29-gxarpqgmndzlfgmdz6iptiqngenzizkl
198M    ./caliper-2.7.0-xzuomtzegahoemci6j2a2qin2ce64arl
460K    ./camp-2022.03.0-qifntyle52cwky7oigsrbbqdzwckf5lo
321M    ./conduit-0.8.3-c5tt2kfm4i2xv2gnjo6kzw4vhsx4ahhl
7.1M    ./curl-7.83.0-kvrbcs2fljjddskhyngdtr4xav47jbyn
147M    ./hdf5-1.8.21-upwmnamtykdpb6p2xs5xrsbh4ikvwf3b
9.6M    ./hypre-2.18.2-qcxc4q672fr6psj6rjgyelhyqsp3ktfy
2.6M    ./lua-5.3.5-hwdhlftajbhwz3thlbht3ubnadnxq6xo
3.0M    ./metis-5.1.0-hig6xr3kxwgnvfyklzozjiw4kszcflbu
51M     ./mfem-4.3.0.2-z46huvt7qmhydqx77m426u4bg6w2q54n
27M     ./ncurses-6.2-f6uucrran47uv4hn7if7yy4c4y6vahvl
43M     ./netcdf-c-4.7.4-p4n5jktcjlud2ao3ebgdwdo6vbozovfg
15M     ./openssl-1.1.1q-yl6ofwzgqhqvfuacmpeolcmtrgyyi2xq
6.0M    ./parmetis-4.0.3-na55ekdi4rd6cyhlmrihjd5s3nzfnvjn
6.7M    ./raja-2022.03.0-p5njsvrzcz4hwnvbf26p4pftbbcp4227
52M     ./sundials-5.7.0-t7yrvtd6azxqinlf4bbx6wav7j76os26
7.5M    ./superlu-dist-6.1.1-zsghmirhnc63dbudil3gviti3p5xe7xd
18M     ./umpire-2022.03.1-u3z2gy5oyksjlb2sgwvc4utcprudgszp
1.4G    ./

With changes:

288K    ./adiak-0.2.1-rhxz3pwsr5bpepl2xebukbr626svey5o
20M     ./axom-0.6.1.6-3cm4l4cao454im4jdwib6lzyp7cpt7kw
7.3M    ./blt-0.5.1-7sde54yl6bh46taa4xfujpkiodmxymb5
276K    ./ca-certificates-mozilla-2022-03-29-gxarpqgmndzlfgmdz6iptiqngenzizkl
18M     ./caliper-2.7.0-3q5j7a3wh7bhdb3xoqbxazwyne6jwg4m
460K    ./camp-2022.03.0-qifntyle52cwky7oigsrbbqdzwckf5lo
46M     ./conduit-0.8.3-zjcin7zk4bxgmir6i7forprcqx4hxdzw
7.1M    ./curl-7.83.0-kvrbcs2fljjddskhyngdtr4xav47jbyn
50M     ./hdf5-1.8.21-gynmheuzgocs7llypsdodck6zbus4rlg
9.6M    ./hypre-2.18.2-qcxc4q672fr6psj6rjgyelhyqsp3ktfy
2.6M    ./lua-5.3.5-hwdhlftajbhwz3thlbht3ubnadnxq6xo
3.0M    ./metis-5.1.0-hig6xr3kxwgnvfyklzozjiw4kszcflbu
45M     ./mfem-4.3.0.2-zptbybovgvwxdwz2c65nsbn4j5pbieus
27M     ./ncurses-6.2-f6uucrran47uv4hn7if7yy4c4y6vahvl
19M     ./netcdf-c-4.7.4-yvf2j7nehwl5lfgurk4nsrr7aro5rd5q
15M     ./openssl-1.1.1q-yl6ofwzgqhqvfuacmpeolcmtrgyyi2xq
3.0M    ./parmetis-4.0.3-n247y72vjy4pzjjbolpg2rv4x4puptk6
4.8M    ./raja-2022.03.0-eiqbuomtypydev7sfwfyv5lkojpurrwg
21M     ./sundials-5.7.0-icdllmf5v4rw6nwhocsdmpwv5kq5m6u5
4.0M    ./superlu-dist-6.1.1-e5kaxndlivlu5iwognsv4o5kqpcqgpfq
3.4M    ./umpire-2022.03.1-j4dhwmgd2ih2npafbgjbwuaoc4a7avob
302M    ./

[white238]

Just tested the shared variant on toss3 and it now passes all the tests.