`readraster(url)` works on mac but not linux (GDALError (CE_Failure, code 4))

[alex-s-gardner]

using ArchGDAL v0.10.8 & GDAL v1.10.0 on a mac and linux machine I am able to read a file on the mac machine but not the linux server:

On mac

import ArchGDAL as AG

url = "https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif"
dataset = AG.readraster("/vsicurl/$url")

GDAL Dataset (Driver: GTiff/GeoTIFF)
File(s): 
  /vsicurl/https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif

Dataset (width x height): 43200 x 21600 (pixels)
Number of raster bands: 1
  [GA_ReadOnly] Band 1 (Gray): 43200 x 21600 (Float32)


"https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif"

GDAL Dataset (Driver: GTiff/GeoTIFF)
File(s): 
  /vsicurl/https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif

Dataset (width x height): 43200 x 21600 (pixels)
Number of raster bands: 1
  [GA_ReadOnly] Band 1 (Gray): 43200 x 21600 (Float32)

same command on on a linux server:

ERROR: GDALError (CE_Failure, code 4):
        `/vsicurl/https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif' does not exist in the file system, and is not recognized as a supported dataset name.

Stacktrace:
 [1] maybe_throw()
   @ GDAL ~/.julia/packages/GDAL/8oAvY/src/error.jl:42
 [2] aftercare
   @ ~/.julia/packages/GDAL/8oAvY/src/error.jl:59 [inlined]
 [3] gdalopenex(pszFilename::String, nOpenFlags::Int64, papszAllowedDrivers::Ptr{…}, papszOpenOptions::Ptr{…}, papszSiblingFiles::Ptr{…})
   @ GDAL ~/.julia/packages/GDAL/8oAvY/src/libgdal.jl:7211
 [4] #read#79
   @ ~/.julia/packages/ArchGDAL/ujstt/src/dataset.jl:630 [inlined]
 [5] read
   @ ~/.julia/packages/ArchGDAL/ujstt/src/dataset.jl:623 [inlined]
 [6] readraster(s::String)
   @ ArchGDAL ~/.julia/packages/ArchGDAL/ujstt/src/raster/array.jl:138
 [7] top-level scope
   @ ~/Documents/GitHub/Altim.jl/src/junk1.jl:16
Some type information was truncated. Use `show(err)` to see complete types.

[evetion]

Works fine on my Linux machine. Are you sure your cluster can access the internet? Maybe debug with

using Downloads
Downloads.request(url, method="HEAD")

or similar?

[alex-s-gardner]

hmmm.. Downloads seems to work fine:

using Downloads
Downloads.request(url, method="HEAD")

Response("https", "https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif", 200, "HTTP/2 200", ["content-type" => "binary/octet-stream", "content-length" => "426507263", "date" => "Mon, 05 May 2025 20:04:59 GMT", "x-amz-replication-status" => "COMPLETED", "last-modified" => "Mon, 29 Jul 2024 18:36:54 GMT", "etag" => "\"1c7079eeb4741890c8dda8e9c751973d\"", "x-amz-server-side-encryption" => "AES256", "x-amz-version-id" => "HxaRMGbkXBw.O72_znbYO4JyvRWj4pKi", "accept-ranges" => "bytes", "server" => "AmazonS3", "x-cache" => "Miss from cloudfront", "via" => "1.1 44852d35e132e198a9d9b6c7289e3cd0.cloudfront.net (CloudFront)", "x-amz-cf-pop" => "LAX54-P1", "x-amz-cf-id" => "ois_jK-TBSsw_hkrbwU9amWGiXfAatXfuzOvPngPPnQbPjFXlOSN6Q==", "x-xss-protection" => "1; mode=block", "x-frame-options" => "SAMEORIGIN", "referrer-policy" => "strict-origin-when-cross-origin", "x-content-type-options" => "nosniff", "strict-transport-security" => "max-age=31536000"])

and this works:

tmpfile = tempname()*".tif"
Downloads.download(url, tmpfile)
dataset = AG.readraster(tmpfile)

[evetion]

Odd. You might want to start your Julia with CPL_CURL_VERBOSE=TRUE in the env, that will get you a lot of debug info on the attempted download.

[alex-s-gardner]

Thanks for the suggestion.. seems like a memory allocation error.

* Couldn't find host data.ghg.center in the .netrc file; using defaults
* Host data.ghg.center:443 was resolved.
* IPv6: 2600:9000:2202:7a00:11:4f0a:f540:93a1, 2600:9000:2202:fe00:11:4f0a:f540:93a1, 2600:9000:2202:e000:11:4f0a:f540:93a1, 2600:9000:2202:8600:11:4f0a:f540:93a1, 2600:9000:2202:400:11:4f0a:f540:93a1, 2600:9000:2202:b600:11:4f0a:f540:93a1, 2600:9000:2202:7e00:11:4f0a:f540:93a1, 2600:9000:2202:da00:11:4f0a:f540:93a1
* IPv4: 13.226.225.26, 13.226.225.18, 13.226.225.77, 13.226.225.91
*   Trying 13.226.225.26:443...
* Connected to data.ghg.center (13.226.225.26) port 443
* Error reading ca cert file /usr/lib/ssl/certs - mbedTLS: (-0x3F80) PK - Memory allocation failed
* Closing connection
ERROR: GDALError (CE_Failure, code 1):
        Error reading ca cert file /usr/lib/ssl/certs - mbedTLS: (-0x3F80) PK - Memory allocation failed

Stacktrace:
 [1] maybe_throw()
   @ GDAL ~/.julia/packages/GDAL/8oAvY/src/error.jl:42
 [2] aftercare
   @ ~/.julia/packages/GDAL/8oAvY/src/error.jl:59 [inlined]
 [3] gdalopenex(pszFilename::String, nOpenFlags::Int64, papszAllowedDrivers::Ptr{…}, papszOpenOptions::Ptr{…}, papszSiblingFiles::Ptr{…})
   @ GDAL ~/.julia/packages/GDAL/8oAvY/src/libgdal.jl:7211
 [4] #read#79
   @ ~/.julia/packages/ArchGDAL/ujstt/src/dataset.jl:630 [inlined]
 [5] read
   @ ~/.julia/packages/ArchGDAL/ujstt/src/dataset.jl:623 [inlined]
 [6] readraster(s::String)
   @ ArchGDAL ~/.julia/packages/ArchGDAL/ujstt/src/raster/array.jl:138
 [7] top-level scope
   @ ~/Documents/GitHub/Altim.jl/src/junk1.jl:16
Some type information was truncated. Use `show(err)` to see complete types.

[alex-s-gardner]

strange thing is when I download using curl at the terminal I don't have any issues:

➜  ~ curl --output junk.tif "https://data.ghg.center/sedac-popdensity-yeargrid5yr-v4.11/gpw_v4_population_density_rev11_2020_30_sec_2020.tif"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  406M  100  406M    0     0  69.1M      0  0:00:05  0:00:05 --:--:-- 78.1M

[alex-s-gardner]

and this works:

run(`$(GDAL.gdalinfo_path()) /vsicurl/$url`)

But not

dataset = AG.readraster("/vsicurl/$url")

Also skipping SSL fixes the problem but I don't think this is good practice

ENV["GDAL_HTTP_UNSAFESSL"]="YES"

[felixcremer]

~Does AG.gdalinfo work?~ That won't work, because this function works on the dataset not on the path.

[evetion]

The error is * Error reading ca cert file /usr/lib/ssl/certs - mbedTLS: (-0x3F80) PK - Memory allocation failed. So I guess you have an ENV variable set to that location, like SSL_CERT_FILE or SSL_CERT_DIR. Remove/emtpy them, or if the first exists, try setting it to an actual file (like a ca-certificates.crt in there, instead of the folder).

[alex-s-gardner]

Thank for all the help @evetion, you where right!

setting the new paths to:

ENV["SSL_CERT_FILE"] = "/etc/ssl/certs/ca-certificates.crt"
ENV["SSL_CERT_DIR"] = "/etc/ssl/certs/"

did the job.