NAS Parallel Benchmarks

[wadudmiah]

Hi,

I built the BT and CG benchmarks from the NAS parallel benchmark suite (https://www.nas.nasa.gov/software/npb.html) with AOMP 15.0 and the BT benchmark is showing 0.00 Mop/s (mega operations per second). The CG benchmark exits with this error message:

 internal error in sparse: i=         7295
Warning: ieee_inexact is signaling
FORTRAN STOP

Could you please build the NAS parallel benchmarks and try to run them yourself? This might indicate an error in the compiler. The NAS Parallel Benchmarks works fine with AOCC, GCC and Intel, so I'm more inclined to think there is an issue with AOMP.

Regards,

[ronlieb]

please provide details on how you compiled and ran the BT and CG benchmarks. scripts. makefiles, compiler options etc...

[wadudmiah]

The only compilation flags that I have used are -O3 -fopenmp which are specified in config/make.def, and the build command used was make <bt|cg> CLASS=A. When running it, I simply used OMP_NUM_THREADS=2 ./bt.A.x, so nothing elaborate.

[Lynd98]

Hi wadudmiah,

Works for me. The PATH and LD_LIBRARY_PATH is dependent on your system and ROCm version. Try "find /opt -name clang" to see what is available.

Check that your stack size is unlimited. Try "ulimit" to see your stacksize, "ulimit -s unlimited" to set it to unlimited.

Lynd

// NAS Parallel Benchmarks BT and CG //

$ wget https://www.nas.nasa.gov/assets/npb/NPB3.4.2.tar.gz $ tar xvf NPB3.4.2.tar.gz $ cd NPB3.4.2 $ cd NPB3.4-OMP $ export PATH=/opt/rocm-5.0.2/llvm/bin:$PATH $ export LD_LIBRARY_PATH=/opt/rocm-5.0.2/llvm/lib:$LD_LIBRARY_PATH $ export OMP_NUM_THREADS=4 $ <copy make.def.template to make.dir, see changes below> $ make BT CLASS=A $ ./bin/bt.A.x $ make CG CLASS=A $ ./bin/cg.A.x

NAS Parallel Benchmarks (NPB3.4-OMP) - CG Benchmark

Size: 14000 Iterations: 15 Number of available threads: 4

Initialization time = 0.107 seconds

iteration ||r|| zeta 1 0.25635941551927E-12 19.9997581277040 2 0.25605704140454E-14 17.1140495745506 3 0.25421676709679E-14 17.1296668946143 4 0.25795666258264E-14 17.1302113581192 5 0.25138229996430E-14 17.1302338856353 6 0.25150840725521E-14 17.1302349879482 7 0.25055122505355E-14 17.1302350498916 8 0.25305434332939E-14 17.1302350537510 9 0.24821350842106E-14 17.1302350540101 10 0.24718579794509E-14 17.1302350540284 11 0.24462039179220E-14 17.1302350540298 12 0.24991619975896E-14 17.1302350540299 13 0.23986004601117E-14 17.1302350540299 14 0.24038699962790E-14 17.1302350540299 15 0.24024635871460E-14 17.1302350540299 Benchmark completed VERIFICATION SUCCESSFUL Zeta is 0.1713023505403E+02 Error is 0.5247076633225E-13

CG Benchmark Completed. Class = A Size = 14000 Iterations = 15 Time in seconds = 0.16 Total threads = 4 Avail threads = 4 Mop/s total = 9073.64 Mop/s/thread = 2268.41 Operation type = floating point Verification = SUCCESSFUL Version = 3.4.2 Compile date = 18 Aug 2022

Compile options: FC = flang FLINK = $(FC) F_LIB = (none) F_INC = (none) FFLAGS = -O3 -fopenmp FLINKFLAGS = $(FFLAGS) RAND = randi8

Please send all errors/feedbacks to:

NPB Development Team [email protected]

// Changes to make.def //

$ diff make.def.template make.def 32c32 < FC = gfortran

FC = flang 38c38 < #---------------------------------------------------------------------------

---

#-------------------------------------------------------------------------- 78c78 < CC = gcc

---

CC = clang 123c123 < UCC = gcc

---

UCC = clang

[wadudmiah]

Hi @Lynd98, cheers for following it up. The stack size is unlimited:

$ ulimit -s
unlimited

I don't think I have ROCm installed - I just downloaded the RPM aomp_CENTOS_7-15.0-3.x86_64.rpm and extracted all the files. These are the directories in the root directory of the RPM:

amdgcn  examples  include  lib-debug  libexec      roctracer
bin     hip       lib      lib64      rocprofiler  share

both the lib and lib64 directories are in the LD_LIBRARY_PATH and the bin directory is in the PATH variable.

[wadudmiah]

ldd on the binary shows:

$ ldd bin/bt.A.x 
        linux-vdso.so.1 =>  (0x00007ffc39d7f000)
        libpgmath.so => /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/lib/libpgmath.so (0x00002b06561ab000)
        libflang.so => /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/lib/libflang.so (0x00002b06563dd000)
        libflangrti.so => /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/lib/libflangrti.so (0x00002b0655fab000)
        libomp.so => /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/lib/libomp.so (0x00002b0655fba000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b06567ee000)
        librt.so.1 => /lib64/librt.so.1 (0x00002b0656af0000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b0656cf8000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b0656f14000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00002b06572e2000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b06575ea000)
        libquadmath.so.0 => /lib64/libquadmath.so.0 (0x00002b0657800000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b0657a3c000)
        /lib64/ld-linux-x86-64.so.2 (0x00002b0655f87000)

so all the dynamic libraries are resolved.

[Lynd98]

Please send me the output to "which flang" and "flang --version".

[wadudmiah]

$ module load aomp $ which flang /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/bin/flang $ flang --version AOMP_STANDALONE_15.0-3 flang-new version 15.0.0 (/home/release/git/aomp15.0/llvm-project/clang b745843ebcb77f55de887b5741197184e7d0dcbd) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/bin $ echo $PATH /local/software/aomp/aomp-15.0/usr/lib/aomp_15.0-3/bin:[other paths]

[Lynd98]

I see the following:

lstringe@dell-rack-14:~/NASParallelBenchmarks/NPB3.4.2/NPB3.4-OMP/bin$ ldd bt.A.x linux-vdso.so.1 (0x00007ffe49f9b000) libpgmath.so => /opt/rocm-5.1.0/llvm/lib/libpgmath.so (0x000014e6d0a0e000) libflang.so => /opt/rocm-5.1.0/llvm/lib/libflang.so (0x000014e6d0611000) libflangrti.so => /opt/rocm-5.1.0/llvm/lib/libflangrti.so (0x000014e6d0602000) libomp.so => /opt/rocm-5.1.0/llvm/lib/libomp.so (0x000014e6d0514000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014e6d03ac000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x000014e6d039f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014e6d037c000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014e6d018a000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000014e6cffa8000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014e6cff8d000) libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x000014e6cff43000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014e6cff3b000) /lib64/ld-linux-x86-64.so.2 (0x000014e6d0c44000)

I'm getting libraries from "/lib/x86_64-linux-gnu" (say for example: libm.so.6) while your libraries are coming from "/lib64". I don't know if that is the difference. If you have "/lib/x86_64-linux-gnu" (or something similar), please place that before "/lib64" in your env. var.: LD_LIBRARY_PATH and give that a try.

[Lynd98]

This is a bug. Fails at -O3 and -O2 with aomp_15.0-3. Works OK at -O1 with aomp_15.0-3. Works OK with -O3 and ROCm 5.1.0.

[ronlieb]

short term workaround add this flag, we are internally working the compiler issue -ffunc-args-alias

[ronlieb]

fix applied to our flang repo, should be in next aomp release, sometime towards end of the month of Oct. commit 43410ae7fe0a3260e39eca362ffa746d75020099 (origin/aomp-dev, aomp-dev) Author: Ron Lieberman [email protected] Date: Mon Oct 3 12:46:27 2022 -0500

[SWDEV-352861] aliasing issue, NAS OMP CG, aomp issue 411

[gregrodgers]

Check if this is fixed in 16.0-1 and close. Otherwise, fix this in 16.0-2

[Lynd98]

Fixed in 16.0-1, will close.