Odd ncdump + hdf4 behavior on macOS

[mathomp4]

All, I'm a bit baffled by this and I need some help from the gurus with an oddity reported by a colleague.

I build netCDF in a set of libraries and as far as I can tell, on both my mac and Linux builds, hdf4 and netcdf-c are all built with the same options. With hdf4 I pass in --disable-netcdf and with netcdf-c I pass in --enable-hdf4. The only big difference is CC, CXX, and FC (clang/gfortran or clang/nagfor on macOS, various on Linux).

But ncdump (and thus I assume all of netcdf?) act differently. On Linux I get:

$ ncdump MOD04_L2.A2017090.0010.051.NRT.hdf | head
netcdf MOD04_L2.A2017090.0010.051.NRT {
dimensions:
	Cell_Along_Swath\:mod04 = 203 ;
	Cell_Across_Swath\:mod04 = 135 ;
	Solution_3_Land\:mod04 = 3 ;
	Solution_1_Land\:mod04 = 2 ;
	Solution_2_Land\:mod04 = 3 ;
	Solution_4_Land\:mod04 = 4 ;
	MODIS_Band_Land\:mod04 = 7 ;
	QA_Byte_Land\:mod04 = 5 ;

on macOS I get:

$ ncdump -hsc MOD04_L2.A2017090.0010.051.NRT.hdf | head
netcdf MOD04_L2.A2017090.0010.051.NRT {
dimensions:
	Cell_Along_Swath\:mod04 = 18446744072820359168 ;
	Cell_Across_Swath\:mod04 = 18446744071679508480 ;
	Solution_3_Land\:mod04 = 50331648 ;
	Solution_1_Land\:mod04 = 33554432 ;
	Solution_2_Land\:mod04 = 50331648 ;
	Solution_4_Land\:mod04 = 67108864 ;
	MODIS_Band_Land\:mod04 = 117440512 ;
	QA_Byte_Land\:mod04 = 83886080 ;

Now this file is around 500 kiB so I'm guessing Linux has it right. (The macOS one wouldn't fit in the known universe probably.)

Does anyone know why this is happening? I freely admit it might be "You built hdf4 wrong with clang" or "on macOS" as, well, I never use hdf4 files myself (only netcdf4) so it's possible I've been doing this wrong for years.

[DennisHeimbigner]

Note the followingfor the first dimension:

18446744072820359168 = 0xffffffffcb000000 203 = 0xcb

It appears that OSX is sign extending the true value (203) but printing it as unsigned. Also do you know if the machine(s) you using are big-endian?

[mathomp4]

@DennisHeimbigner Well that is...weird. I mean, one machine is NCCS Discover (SLES12) and the other is a MacBook Pro (macOS Ventura). Neither are big-endian at all, and my Baselibs has no flags for endianness conversion.

I guess I'll stare at the macOS config.log for hdf4 (and dependent libraries like szip and jpeg) and see if it's somehow getting the wrong endianness?

[mathomp4]

Huh. Well from the hdf4 (and hdf5 and netcdf) configure:

checking whether byte ordering is bigendian... no

so autotools is quite satisified this is a big-endian system.

My colleague whipped up a code that reads the HDF4 file via netcdf (Fortran in this case) and on Linux:

+ ./test.x
  filename=MOD04_L2.A2017090.2000.051.NRT.hdf
 fileName=MOD04_L2.A2017090.2000.051.NRT.hdf
 ncid=       65536
 lon_name=Cell_Along_Swath:mod04
 ck step 1
 ck step 2
  nlat, nlon                             135                             203

but on macOS:

+ ./test.x
  filename=MOD04_L2.A2017090.2000.051.NRT.hdf
 fileName=MOD04_L2.A2017090.2000.051.NRT.hdf
 ncid=       65536
 lon_name=Cell_Along_Swath:mod04
 ck step 1
 ck step 2
  nlat, nlon                     -2030043136                      -889192448

Just...bizarre.

I even tried using -fconvert=big-endian on the macOS one to see if that changed anything, but no difference (as expected).

[mathomp4]

Well, I managed to show oddities with hdp and asked the HDF Group here:

https://forum.hdfgroup.org/t/weird-hdf4-behavior-on-macos/11603