pyproj icon indicating copy to clipboard operation
pyproj copied to clipboard

Published 20 hours ago • verified publisher icon pyproj4

Randomly appearing bug with multi-threaded Transformer in versions >= 3.7.0

Open odhondt opened this issue 5 months ago • 13 comments

Code Sample, a copy-pastable example if possible

import numpy as np
from pyproj import Transformer
from joblib import Parallel, delayed
import time

# Generate test data
N = 1000
lat = np.random.uniform(-90, 90, N)
lon = np.random.uniform(-180, 180, N)
alt = np.random.uniform(0, 1000, N)

composite_crs = "EPSG:4326+3855"
ECEF_crs = "EPSG:4978"
chunk_size = 128

# Split into chunks
chunks = [
    (lon[i:i+chunk_size], lat[i:i+chunk_size], alt[i:i+chunk_size])
    for i in range(0, N, chunk_size)
]


def run_single_threaded():
    tf = Transformer.from_crs(composite_crs, ECEF_crs, always_xy=True)
    results = tf.transform(lon, lat, alt)
    return results

def run_multi_threaded():
    def transform_chunk(lon_chunk, lat_chunk, alt_chunk):
        tf = Transformer.from_crs(composite_crs, ECEF_crs, always_xy=True)
        return tf.transform(lon_chunk, lat_chunk, alt_chunk)

    results = Parallel(n_jobs=-1, prefer="threads")(
        delayed(transform_chunk)(lon, lat, alt) for lon, lat, alt in chunks
    )
    return np.concatenate([r[0] for r in results]), np.concatenate([r[1] for r in results]), np.concatenate([r[2] for r in results])

x1, y1, z1 = run_single_threaded()
x2, y2, z2 = run_multi_threaded()

# --- Check results
def check_close(a, b, label):
    if np.allclose(a, b, atol=1e-6):
        print(f"{label:20s}: Match")
    else:
        print(f"{label:20s}: MISMATCH")

check_close(x1, x2, "Single vs Threads (X)")
check_close(y1, y2, "Single vs Threads (Y)")
check_close(z1, z2, "Single vs Threads (Z)")

Problem description

After switching to 3.7.0, converting coordinates to a composite CRS EPSG:4326+3855 while multi-threading sometimes throws results with chunks of inf values.

Here is the output of the above code with versions >= 3.7.0

TIFFFillTile: Read error at row 2816, col 2816, tile 54; got 0 bytes, expected 124661. 
TIFFFillTile: Read error at row 4352, col 0, tile 296; got 0 bytes, expected 137296. 
TIFFFillTile: Read error at row 4352, col 0, tile 317; got 0 bytes, expected 166622. 
TIFFFillTile: Read error at row 2560, col 2560, tile 351; got 0 bytes, expected 160283.
 
Single vs Threads (X): MISMATCH 
Single vs Threads (Y): MISMATCH 
Single vs Threads (Z): MISMATCH

Note that is does not occur every time but randomly.

Expected Output

With versions 3.6.1 it seems to be always running as expected:

Single vs Threads (X): Match 
Single vs Threads (Y): Match 
Single vs Threads (Z): Match

Environment Information

pyproj info:
    pyproj: 3.7.0
PROJ (runtime): 9.5.1
PROJ (compiled): 9.5.0
  data dir: /root/micromamba/envs/eo_tools/share/proj
user_data_dir: /root/.local/share/proj
PROJ DATA (recommended version): 1.20
PROJ Database: 1.4
EPSG Database: v11.022 [2024-11-05]
ESRI Database: ArcGIS Pro 3.4 [2024-11-04]
IGNF Database: 3.1.0 [2019-05-24]

System:
    python: 3.12.10 | packaged by conda-forge | (main, Apr 10 2025, 22:21:13) [GCC 13.3.0]
executable: /root/micromamba/envs/eo_tools/bin/python
   machine: Linux-6.8.0-59-generic-x86_64-with-glibc2.36

Python deps:
   certifi: 2025.4.26
    Cython: None
setuptools: 80.9.0
       pip: 25.1.1

Installation method

conda (actually micromamba) using the conda-forge repo.

Conda environment information (if you installed with conda):


Environment (conda list): 
$ conda list proj


  proj    9.5.1    h0054346_0       conda-forge
  pyproj  3.7.0    py312he630544_0  conda-forge

Details about conda and system ( conda info ): 
$ conda info


       libmamba version : 2.1.1
     micromamba version : 2.1.1
           curl version : libcurl/8.13.0 OpenSSL/3.5.0 zlib/1.3.1 zstd/1.5.7 libssh2/1.11.1 nghttp2/1.64.0
     libarchive version : libarchive 3.7.7 zlib/1.3.1 bz2lib/1.0.8 libzstd/1.5.7
       envs directories : /root/micromamba/envs
          package cache : /root/micromamba/pkgs
                          /root/.mamba/pkgs
            environment : eo_tools (active)
           env location : /root/micromamba/envs/eo_tools
      user config files : /root/.mambarc
 populated config files : 
       virtual packages : __unix=0=0
                          __linux=6.8.0=0
                          __glibc=2.36=0
                          __archspec=1=x86_64_v3
               channels : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
       base environment : /root/micromamba
               platform : linux-64

odhondt avatar Jun 03 '25 10:06 odhondt

Mind debugging internal PROJ? See: https://pyproj4.github.io/pyproj/stable/advanced_examples.html#debugging-internal-proj

snowman2 avatar Jun 04 '25 00:06 snowman2

Sure! I have added:

import os
import logging

os.environ["PROJ_DEBUG"] = "2"

console_handler = logging.StreamHandler()
formatter = logging.Formatter("%(levelname)s:%(message)s")
console_handler.setFormatter(formatter)
logger = logging.getLogger("pyproj")
logger.addHandler(console_handler)
logger.setLevel(logging.DEBUG)
logging.basicConfig(format="%(levelname)s:%(message)s", level=logging.DEBUG)

at the beginning of my code and this is what I get:

DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(egm08_25.gtx): call fopen(/root/.local/share/proj/egm08_25.gtx) - failed
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_DEBUG: Using https://cdn.proj.org/us_nga_egm08_25.tif
DEBUG:PROJ_ERROR: Cannot take exclusive lock on /root/.local/share/proj/cache.db
DEBUG:PROJ_ERROR: Cannot take exclusive lock on /root/.local/share/proj/cache.db
TIFFFillTile: Read error at row 3840, col 3840, tile 230; got 0 bytes, expected 152183.
DEBUG:PROJ_ERROR: Cannot take exclusive lock on /root/.local/share/proj/cache.db
DEBUG:PROJ_ERROR: Cannot take exclusive lock on /root/.local/share/proj/cache.db
TIFFFillTile: Read error at row 4608, col 256, tile 322; got 0 bytes, expected 153864.

I could push the debug level to 3 or 4 but the ouput gets much longer (>10000 lines). Let me know!

Btw, I forgot to mention I was running this in a docker container 😄

odhondt avatar Jun 04 '25 06:06 odhondt

What happens if you download the grids first and disable PROJ Network?

snowman2 avatar Jun 05 '25 12:06 snowman2

I am not a pyproj specialist, so let me know if it wasn't what you meant, but:

  • I have manually downloaded the grid at https://cdn.proj.org/us_nga_egm08_25.tif
  • copied it in /root/.local/share/proj
  • added set_network_enabled(False) before starting the conversion

The bug seems to disappear:

DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(proj.ini): call fopen(/root/micromamba/envs/eo_tools/share/proj/proj.ini) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz): call fopen(/root/.local/share/proj/Und_min1x1_egm2008_isw=82_WGS84_TideFree.gz) - failed
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: Using coordinate operation Inverse of WGS 84 to EGM2008 height (1) + Conversion from WGS 84 (geog3D) to WGS 84 (geocentric)
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded
DEBUG:PROJ_DEBUG: pj_open_lib(us_nga_egm08_25.tif): call fopen(/root/.local/share/proj/us_nga_egm08_25.tif) - succeeded

and the code output is always:

Single vs Threads (X): Match
Single vs Threads (Y): Match
Single vs Threads (Z): Match

odhondt avatar Jun 05 '25 13:06 odhondt

That was perfect, thank you. If you have a moment to try out your original set up with PROJ NETWORK and pyproj 2.5, that would be helpful.

snowman2 avatar Jun 06 '25 00:06 snowman2

Great! Happy to help.

With pyproj 2.5.0 I am not able to control the network as pyproj.network does not exist. Setting the debug env does not seem to have any effect either. I was only able to remove the grid file and relaunch the script, which worked as expected.

odhondt avatar Jun 06 '25 08:06 odhondt

If you set the environment variable PROJ_NETWORK to ON, it should enable it:

https://proj.org/en/stable/usage/network.html#how-to-enable-network-capabilities

snowman2 avatar Jun 07 '25 01:06 snowman2

Thanks. I have tried with and without network and it didn't change anything on 2.5.0, the outputs always match.

odhondt avatar Jun 09 '25 09:06 odhondt

Hi, to follow up on this, is there a way to pre-download the egm tiles programmatically as a workaround? I'd really like to be able upgrade to the latest version 🙂

odhondt avatar Jun 19 '25 09:06 odhondt

Maybe https://proj.org/en/stable/apps/projsync.html works for you

jjimenezshaw avatar Jun 19 '25 09:06 jjimenezshaw

@jjimenezshaw I'll have a look, thanks!

odhondt avatar Jun 19 '25 10:06 odhondt

See: https://pyproj4.github.io/pyproj/stable/transformation_grids.html

snowman2 avatar Jun 19 '25 17:06 snowman2

It worked. From what I understand, the cache.db (SQLite) file may encounter concurrency issues. For others who might face a similar problem, a workaround is to download the required grids into one of the PROJ data directories before performing the transformation. Thanks again for your help!

odhondt avatar Jun 23 '25 13:06 odhondt

So is the assumption here that there is a race condition regarding downloading of the grids and calculations that use those grids? For example, one thread starts the calculation and realizes it doesn't have the grids so it starts downloading them, another thread starts its calculations and sees that is already "has" the grids, but they are incomplete and an error occurs?

What about pyproj 3.7 triggers this?

Is this something that is expected given the existing thread-safety of pyproj or is this a bug in PROJ? Can/should we get an example of pure PROJ in C++ where this occurs?

I wonder if we can reproduce this in pure python multi-threading (no joblib).

djhoese avatar Aug 04 '25 20:08 djhoese

@odhondt My conda-forge environment (Python 3.12) is using pyproj 3.7.1 and PROJ 9.6.0. Are you able to reproduce this with that version of PROJ? I'm trying your original code and I'm always getting "Match" so far.

Possibly related changes in the PROJ 9.6.0 "NEWS" release notes:

createFromPROJString(): avoid repeated openings of proj.db and proj.ini and lookup of 'epsg' ([#4334](https://github.com/OSGeo/PROJ/issues/4334))

Debugging/internals: append sub-grid index to grid name when opening a multi-grid TIFF file ([#4381](https://github.com/OSGeo/PROJ/issues/4381))

createOperations(): use more appropriate operation when using a 'PROJ {grid_name}' geoid model, based on matching the vertical datum ([#4379](https://github.com/OSGeo/PROJ/issues/4379))

https://proj.org/en/stable/news.html#id59

djhoese avatar Aug 04 '25 21:08 djhoese

@djhoese what you mention could be the reason. How are the different "context" objects (one per thread I hope) managed in pyproj?

jjimenezshaw avatar Aug 04 '25 21:08 jjimenezshaw

I just updated my last comment with notes from PROJ's release notes that may have fixed this.

As far as what context stuff, Alan has to answer that as I've lost track of his changes, fixes, improvements to that kind of stuff.

djhoese avatar Aug 04 '25 21:08 djhoese

I just tried with older proj and I'm still not able to reproduce this with a pure conda-forge environment on bare metal (not a docker container). I'm on PopOS (ubuntu), intel CPU.

$ conda install "proj<9.6.0"
...

The following packages will be DOWNGRADED:

  proj                                     9.6.2-h18fbb6c_1 --> 9.5.1-h0054346_0 

The following packages will be REVISED:

  pyproj                              3.7.1-py312h03c6e1f_1 --> 3.7.1-py312he630544_0 

...


$ for x in {0..10}; do python issue_1499.py; echo $?; done
Single vs Threads (X): Match
Single vs Threads (Y): Match
Single vs Threads (Z): Match
0
Single vs Threads (X): Match
Single vs Threads (Y): Match
Single vs Threads (Z): Match
0
Single vs Threads (X): Match
Single vs Threads (Y): Match
Single vs Threads (Z): Match
0
...


$ env | grep PROJ
PROJ_DATA=/home/davidh/miniforge3/envs/pyproj_thread_debug/share/proj
PROJ_NETWORK=ON


$ pyproj -v
pyproj info:
    pyproj: 3.7.1
PROJ (runtime): 9.5.1
PROJ (compiled): 9.5.1
  data dir: /home/davidh/miniforge3/envs/pyproj_thread_debug/share/proj
user_data_dir: /home/davidh/.local/share/proj
PROJ DATA (recommended version): 1.20
PROJ Database: 1.4
EPSG Database: v11.022 [2024-11-05]
ESRI Database: ArcGIS Pro 3.4 [2024-11-04]
IGNF Database: 3.1.0 [2019-05-24]

System:
    python: 3.12.11 | packaged by conda-forge | (main, Jun  4 2025, 14:45:31) [GCC 13.3.0]
executable: /home/davidh/miniforge3/envs/pyproj_thread_debug/bin/python
   machine: Linux-6.12.10-76061203-generic-x86_64-with-glibc2.35

Python deps:
   certifi: 2025.8.3
    Cython: None
setuptools: 80.9.0
       pip: 25.2

How often is this failing for you @odhondt?

djhoese avatar Aug 05 '25 01:08 djhoese

I am guessing that the issue is in PROJ and #1133 uncovered the issue. However, I could be mistaken. I haven't had time to dig into the issue or raise an upstream issue.

snowman2 avatar Aug 05 '25 02:08 snowman2