ESCOMP
Clarification of use of various (external) libraries for cesm-coupling
Below is based on my non-expert understanding, apologize in advance if not accurate.
Currently, the Makefile for cesm-coupling branch assumes the structure libraries that are prepared by ./manage_externals/checkout_externals. This perhaps results in mizuRoute Makefile, or the Makefile for libraries, assuming a relative path to each of the needed files during compilation. Tracking each of the libraries also gets very complex across many existing Makefiles.
A way forward can be to specify all the packages, similar to netcdf for example, in mizuRoute Makefile. This link can be for gptl or parallelio. The Makefile can be directly pointed at the location of the libraries if they are built prior to the compilation of mizuRoute. This also makes it perhaps easier to link the Makefile to libraries prepared by ./manage_externals/checkout_externals.
Definition of done:
- [x] Remove uneeded externals: cime, mct, mpi-serial
- [ ] Figure out how to build PIO with GPTL on for easybuild
- [ ] Same for spack?
- [ ] Remove isGPTL from the Makefile -- you really need the version in PIO that has the Fortran layer
- [ ] Hardcode GPTL on in the route/build/lib/Makefile
- [ ] Add a note to the Makefile, that you need to build with GPTL on, so use: PIO_ENABLE_TIMING=ON when using an - external PIO library
- [ ] Add similar notes to other places that talk about the standalone build, for example the user's guide, and the README file in route/build
- [ ] Add a graceful error check to the Makefile that autodetects if the external PIO library was built with GPTL or not
Perhaps a bit of more explanation of what I am after; I would like to see the path from pre-built libraries in the mizuRoute Makefile, sth like:
PNETCDF = path/to/parallel/netcdf # which includes lib/libpnetcdf.a
NETCDFC = path/to/netcdf/c # which includes lib/libnetcdf.a
NETCDFF = path/to/netcdf/fortran # which includes lib/libnetcdff.a
GPTL = path/to/gptl # which includes lib/libgptl.a and lib/libgptlf.a
PIO = path/to/parallel/io # which includes lib/libpioc.a and lib/libpiof.a
in case the user sets the flags to true or yes for PIO or GPTL and the paths are not provided, makefile can be directed to externals. If the paths are provided, it is expected that mizuRoute src is built against the provided pre-built libraries.
Would that be a feasible option?
Hi Erik (@ekluzek), any comments on this?? maybe we will need to talk to Brian (Dobbin)??
More information on what I am currently trying to achieve. I am trying to create a dockerfile with a spack environment inside it. I managed to clone and build the mizuRoute main branch with some minor modifications to the makefile, however, I am not successful with the cesm-coupling branch. I highly suspect this is because of the interconnected path for the libraries that are populated by checkout_extrenals and need to be built during the compilation of mizuRoute. The dockerfile looks like this:
# Use Ubuntu as the base image
FROM ubuntu:20.04 as builder
# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y \
build-essential \
ca-certificates \
coreutils \
curl \
environment-modules \
gfortran \
gpg \
lsb-release \
python3 \
python3-distutils \
python3-pip \
python3-venv \
unzip \
zip \
git \
nano \
wget
# Set up Spack environment
RUN rm -rf /opt/spack-environment
RUN mkdir -p /opt/spack-environment
RUN (echo "spack:" \
&& echo " specs:" \
&& echo " - zlib" \
&& echo " - openmpi" \
&& echo " - netcdf-c" \
&& echo " - netcdf-fortran" \
&& echo " - parallel-netcdf" \
&& echo " - parallelio" \
&& echo " - gptl" \
&& echo " concretizer:" \
&& echo " unify: true" \
&& echo " config:" \
&& echo " install_tree: /opt/software" \
&& echo " view: /opt/view") > /opt/spack-environment/spack.yaml
# Install Spack and the specs
RUN rm -rf /opt/spack && \
git clone -c feature.manyFiles=true https://github.com/spack/spack.git /opt/spack && \
. /opt/spack/share/spack/setup-env.sh && \
cd /opt/spack-environment && \
spack env activate . && spack install --fail-fast && spack gc -y
# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
xargs file -i | \
grep 'charset=binary' | \
grep 'x-executable\|x-archive\|x-sharedlib' | \
awk -F: '{print $1}' | xargs strip -s
# Set up environment modifications
RUN . /opt/spack/share/spack/setup-env.sh && \
spack env activate --sh -d /opt/spack-environment >> /etc/profile.d/z10_spack_environment.sh
# Bare OS image to run the installed executables
FROM ubuntu:20.04
COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /opt/spack /opt/spack
COPY --from=builder /usr /usr
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
# Activate Spack environment and load packages
. /etc/profile.d/z10_spack_environment.sh && \
. /opt/spack/share/spack/setup-env.sh && \
spack env activate /opt/spack-environment -p
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
after building the image and run the container:
docker build -t spacktest .
docker run -it spacktest
or restart existing container after running the target container in docker desktop
docker exec -it <hash> bash
after running the docker container, activating the spack env
. /etc/profile.d/z10_spack_environment.sh
. /opt/spack/share/spack/setup-env.sh
spack env activate /opt/spack-environment -p
and try to see where the packages are installed
spack find -p netcdf-fortran
inside the library can be check by cd to the path of spack build location
cd $(spack location -i netcdf-c)/lib
I believe if the path are separated in the makefile I should be able to first manually link each of the built packages to the makefile and hopefully compile mizuRoute inside the container (similar to the step I took for compiling the main branch).
Ok, I tested compiling with system installed parallelio, and get it compiled.
The timing functions used in mizuRoute does not directly use the GPTL timing functions, but use the module built in parallelio that wrap GPTL functions (the module called perf_mod, and functions like t_prf, t_finalizedf). So If I try to use the system installed gptl, the syntax is not matched and is not be compiled. So gptl library needs to be built with parallelio, then mizuRoute should be compiled.
For me, I cannot re-build parallelio with gptl on NCAR HPC (or maybe I could ask if it exists), so I commented out all the timing functions from perf_mod, and excluded gptl, then compiled.
So I changed the makefile this portion (and removed yes from isGPTL =)
223 #========================================================================
224 # External libaries that might need to be built
225 #========================================================================
226 #LIBDIR = $(F_MASTER)build/lib
227 #PIOLIBDIR = $(LIBDIR)/piolib/lib
228 #PIOINCDIR = $(LIBDIR)/piolib/include
229 #PIOLIB = $(PIOLIBDIR)/libpiof.a $(PIOLIBDIR)/libpioc.a
230 #MPISERLIBDIR = $(LIBDIR)/mpi-seriallib
231 #MPISERLIB = $(MPISERLIBDIR)/libmpi-serial.a
232
233 PIOLIBDIR= $(NCAR_ROOT_PARALLELIO)
234 PIOINCDIR= $(NCAR_ROOT_PARALLELIO)/include
235 PIOLIB = $(PIOLIBDIR)/lib/libpiof.so $(PIOLIBDIR)/lib/libpioc.so
236 GPTLLIBDIR=$(NCAR_ROOT_GPTL)
237 GPTLINCDIR=$(NCAR_ROOT_GPTL)/include
238 GPTLLIB =$(GPTLLIBDIR)/lib/libgptl.so
239
240
241 ifeq "$(isPIO)" "yes"
242 EXTINCLUDES += -I$(PIOINCDIR)
243 EXTLIBS += $(PIOLIB)
244 LDFLAGS += -L$(PIOLIBDIR)/lib -lpiof -lpioc
245
246 ifeq "$(isGPTL)" "yes"
247 # LDFLAGS += -lgptl
248 EXTINCLUDES += -I$(GPTLINCDIR)
249 EXTLIBS += $(GPTLLIB)
250 LDFLAGS += -L$(GPTLLIBDIR)/lib -lgptl
251 endif
252 endif
I am not sure what the best way to handle gptl so it works for both stand-alone and cesm (cesm use gptl from parallelio)
#461 removed uneeded externals (mpi-serial, mct, cime), so it clarifies a lot of this. So only PIO is needed now.
The GPTL library is still an issue though because the PIO build needs to have it built with GPTL enabled in order to use it. So building it within the mizuRoute build always works, but using a pre-built library requires that library to have GPTL enabled, and that isn't always the case.
