jayeshkrishna

Just checking, has this issue been fixed? (if not, do you have any approx timeline on when this issue will be fixed?)

Looks like a file access permission issue to me, @sarich can this file permissions be changed to 755 to see if that helps?

ok, the lock files might just be a symptom not the cause of the issue. One thing I do notice is that the filename does not include the full path....

@wlin7 : Do you know how the above filename (v2.LR.historical_0101.eam.i.2015-01-01-00000.nc) gets picked up and why it does not have the full path?

@ndkeen : can you ncdump on the file using the soft link?

Thanks @ndkeen ! We might need a script that scans (weekly? daily?) the file format of files in the inputdata directory to ensure we don't have these issues with NetCDF4...

@sarich had gone through the input files and converted them to 64bit-offset a while back (https://acme-climate.atlassian.net/wiki/spaces/EIDMG/pages/921436224/Netcdf+file+conversion). So you can use the ncks (NetCDF Kitchen sink) command to do the conversion...

Yes, there is still a bug in OpenMPI causing this issue

The OpenMPI devs just mentioned that OpenMPI 4.1.3 (to be released this week) has some fixes that might be relevant to this issue

@dqwu is working on avoiding calls in SCORPIO that generate (cause OpenMPI to generate) these lock files - https://github.com/E3SM-Project/scorpio/pull/472