ERROR 1: PROJ: Cannot take exclusive lock on .local/share/proj/cache.db

[MarcYin]

When import gdal, the PROJ library throws error:

ERROR 1: PROJ: Cannot take exclusive lock on .local/share/proj/cache.db

This only happens when I have multiple computing nodes using the same conda environment

from osgeo import gdal

Problem description

Expected Output

Environment Information

• PROJ version (proj 9.1.1)
• Linux version 3.10.0-1160.92.1.el7.x86_64

Installation method

• mamba

[rouault]

@MarcYin It would be useful if you could provide a minimum Python reproducer with either gdal or pyproj4

You can also try setting the PROJ_LOCK_MAX_ITERS environment variable to a value greater than 30. 30 corresponds to ~ 1 second of retry attempts to get the exclusive lock

[MarcYin]

Hi,

It is a bit python code involves creating VRT from list of tif file to resample them to 10 meters and read:

from osgeo import gdal

url_header = '/vsicurl/https://gws-access.jasmin.ac.uk/public/nceo_isp/S2/30/U/VB/S2A_MSIL1C_20211028T111201_N0301_R137_T30UVB_20211028T131717.SAFE/GRANULE/L1C_T30UVB_A033167_20211028T111452/IMG_DATA/T30UVB_20211028T111201_'
sur_refs = [f'{url_header}{band}_sur.tif' for band in ['B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B11', 'B12']]

# Reflectance
g = gdal.BuildVRT('', sur_refs, separate=True, xRes=10, yRes=10, resampleAlg=gdal.GRIORA_NearestNeighbour)
data = g.ReadAsArray()
print(data.shape)

[rouault]

It is a bit python code involves creating VRT from list of tif file to resample them to 10 meters and read:

that doesn't hit the bug. I guess you need to add some multiprocessing/multithreading on top of that to build a full reproducer

[MarcYin]

Hi,

I was trying to do that but it seems not tigger the error. But I hope a bit more context on this will help to identify the potential issue. The processing involves:

1. reading the Sentinel-2 data using the code I posted above (potential issue here for crossing meridian line?)
2. reading a land cover map in British National Grid projection covering the Sentinel-2 tile with gdalwarp (potential issue here for projecting the British National Grid to the different MGRS tile projection?)

The Sentinel-2 data covers the entire UK, and the reading of data is submitted to different computing nodes sharing the same conda environment.

The error I mentioned above did not trigger all the time, but only occasionally.

Thanks.

[klassenjs]

From [1], it appears that cache.db is used to cache grid files from the CDN.

For multi-node/HPC use, would it be more appropriate to manually installing the grids from https://cdn.proj.org (and thus avoid the need for many requests to the CDN and avoid the lock contention over cache.db)?

[1] https://github.com/OSGeo/PROJ/blob/master/docs/source/usage/network.rst#caching

[jjimenezshaw]

From [1], it appears that cache.db is used to cache grid files from the CDN.

For multi-node/HPC use, would it be more appropriate to manually installing the grids from https://cdn.proj.org (and thus avoid the need for many requests to the CDN and avoid the lock contention over cache.db)?

[1] https://github.com/OSGeo/PROJ/blob/master/docs/source/usage/network.rst#caching

In that case you should download them before, using https://proj.org/en/stable/apps/projsync.html