[Bug] Broken "List available extensions" on Windows

[landam]

Describe the bug

Listing extensions/addons is broken on Windows.

To Reproduce

Open GRASS on Windows (tested with https://grass.osgeo.org/grass80/binary/mswindows/native/WinGRASS-8.0.0-1-Setup.exe) and type:

g.extension -l
ERROR: Download file from <https://grass.osgeo.org/addons/grass8/modules.xml>, failed. File is not on the server or check your internet connection.

But https://grass.osgeo.org/addons/grass8/modules.xml exists...

BTW, 8.0.0RC2 worked without any problem...

Expected behavior

List of extensions/addons printed out :-)

Screenshots

System description (please complete the following information):

• Operating System: Windows
• GRASS GIS version 8.0.0

[tmszi]

I tested it and it works as expected (WinGRASS-8.0.0-1-Setup.exe). Your error message is File is not on the server or check your internet connection.. It looks like you're not connected to the internet (or connection was disabled), or are you using a proxy to connect to the internet and it hasn't been set up?

[neteler]

Is this really a blocker for 8.0.1?

[landam]

I will check today on different machine.

[landam]

Closing as invalid. It was apparently related to a problem with internet connection on my laptop. Sorry for the noise.

[hellik]

now tested with https://grass.osgeo.org/grass80/binary/mswindows/native/WinGRASS-8.0.0-2-Setup.exe while opening g.extension GUI:

Exception in thread
Thread-25
:
Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS
8.0\Python39\lib\threading.py", line 954, in
_bootstrap_inner

self.run()
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\core\gthread.py", line 154, in __run

self.__run_backup()
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\core\gthread.py", line 117, in run

ret = vars()["callable"](*args, **kwds)
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\modules\extensions.py", line 369, in Load

raise GException(_("Unable to load extensions. %s") % msg)
core.gcmd
.
GException
:
Unable to load extensions. Download file from
<https://grass.osgeo.org/addons/grass8/modules.xml>, failed.
File is not on the server or check your internet connection.

[hellik]

g.extension -l                                                                  
List of available extensions (modules):
ERROR: Download file from <https://grass.osgeo.org/addons/grass8/modules.xml>, failed. File is not on the server or check your internet connection.
(Sat Feb 12 21:43:28 2022) Befehl ausgeführt (2 Sek)      

[hellik]

though, the file seems to be on the server:

	logs/	2022-02-12 05:55	-
	modules.xml	2022-02-12 05:59	182K

[DIR] [logs/](https://grass.osgeo.org/addons/grass8/logs/) 2022-02-12 05:55 - [ ] [modules.xml](https://grass.osgeo.org/addons/grass8/modules.xml) 2022-02-12 05:59 182K

[hellik]

a curl download works in the winGRASS console

D:\dl>curl -O https://grass.osgeo.org/addons/grass8/modules.xml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  182k  100  182k    0     0   115k      0  0:00:01  0:00:01 --:--:--  115k

[hellik]

D:\dl>g.extension -l
List of available extensions (modules):
FEHLER: Download file from
        <https://grass.osgeo.org/addons/grass8/modules.xml>, failed. File
        is not on the server or check your internet connection.

as the curl download of the file works, the internet connection is ok.

[petrasovaa]

Can you actually download any addons? I have seen a user struggling with similar problem when downloading extensions, it complains about connection, but that was not the problem. You could try running the g.extension code here from Python shell manually: https://github.com/OSGeo/grass/blob/main/scripts/g.extension/g.extension.py#L243

HEADERS = { "User-Agent": "Mozilla/5.0"}
request = urlrequest.Request("https://grass.osgeo.org/addons/grass8/modules.xml", headers=HEADERS)
urlrequest.urlopen(request)

[tmszi]

@helik I can't reproduce this bug with WinGRASS-8.0.0-2-Setup.exe.

But If I disable internet connection I get error message:

C:\Users\IEUser>g.extension -l
List of available extensions (modules):
ERROR: Download file from
       <https://grass.osgeo.org/addons/grass7/modules.xml>, failed. File is
       not on the server or check your internet connection.

[hellik]

@helik I can't reproduce this bug with WinGRASS-8.0.0-2-Setup.exe.

But If I disable internet connection I get error message:

C:\Users\IEUser>g.extension -l
List of available extensions (modules):
ERROR: Download file from
       <https://grass.osgeo.org/addons/grass7/modules.xml>, failed. File is
       not on the server or check your internet connection.

As mentioned above, the curl download in the same winGRASS 8 session works

[hellik]

Can you actually download any addons? I have seen a user struggling with similar problem when downloading extensions, it complains about connection, but that was not the problem. You could try running the g.extension code here from Python shell manually: https://github.com/OSGeo/grass/blob/main/scripts/g.extension/g.extension.py#L243

HEADERS = { "User-Agent": "Mozilla/5.0"}
request = urlrequest.Request("https://grass.osgeo.org/addons/grass8/modules.xml", headers=HEADERS)
urlrequest.urlopen(request)

Internet connection is up and working, though I can't download any addon via the g.extension GUI

WIill try the code later.

[tmszi]

With this PR #2203 request exceptions are handled in more detail.

[hellik]

Can you actually download any addons? I have seen a user struggling with similar problem when downloading extensions, it complains about connection, but that was not the problem. You could try running the g.extension code here from Python shell manually: https://github.com/OSGeo/grass/blob/main/scripts/g.extension/g.extension.py#L243

HEADERS = { "User-Agent": "Mozilla/5.0"}
request = urlrequest.Request("https://grass.osgeo.org/addons/grass8/modules.xml", headers=HEADERS)
urlrequest.urlopen(request)

tested now in the winGRASS python console:

import xml.etree.ElementTree as etree
from distutils.dir_util import copy_tree
from six.moves.urllib import request as urlrequest
from six.moves.urllib.error import HTTPError, URLError
from six.moves.urllib.parse import urlparse
HEADERS = { "User-Agent": "Mozilla/5.0"}
request = urlrequest.Request("https://grass.osgeo.org/addons/grass8/modules.xml", headers=HEADERS)
urlrequest.urlopen(request)
Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1008, in _send_output
    self.send(msg)
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 948, in send
    self.connect()
  File "C:\OSGeo4W\apps\Python39\lib\http\client.py", line 1422, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\OSGeo4W\apps\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: A failure in the SSL library occurred (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\OSGeo4W\apps\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error A failure in the SSL library occurred (_ssl.c:1129)>

[tmszi]

urllib.error.URLError: <urlopen error A failure in the SSL library occurred (_ssl.c:1129)>

Seems there is a problem with verification SSL certificate (on the client side) compared to the existing list of certificates in your Windows OS. I found some info about Managing Trusted Root Certificates in Windows 10 and 11, maybe this list need update.

[ninsbl]

This works for me with both GRASS GIS 7.8 and 8.0 installed via OSGeo4W.... So, this may be closed, at least it should not be a blocker!?!

[hellik]

See also a recent post in the dev ML

[GRASS-dev] addons list empty

I've tested it on several win 10 boxes. In some of them this issue occurs, in some of them not.

It seems to be issue found by @tmszi

Not sure how to solve this. At least some of our winGRASS users seems to have the same problem.

[hellik]

Manually update win 10 SSL certs isn't an option. It could also be an issue on the server side. IIRC there have been some SSL issues on Martin's server in the past.

[tmszi]

I've tested it on several win 10 boxes. In some of them this issue occurs, in some of them not.

Could you please check if ISRG Root X1 certificate exists on the computers with this issue?

Result of SSL server test and Let's Encrypt ceritificate compatibility.

1. Launch PowerShell
2. Run command: Get-Childitem cert:\LocalMachine\root |format-list
3. See if ISRG Root X1 certificate exists in the list

[neteler]

Is this a GRASS GIS or a Windows problem?

[hellik]

Is this a GRASS GIS or a Windows problem?

See

https://lists.osgeo.org/pipermail/grass-dev/2022-February/095584.html

And the mentioned github issue with some coding workaround in there for more background.

For me it's still not clear, why it works on some winboxes and on some not.

AFAICT there are winGRASS users out there with this issue of not able to install addons.

Is there a way to check if outdated certificates on the download server side can be excluded as reason of this issue?

[tmszi]

And the mentioned github issue with some coding workaround in there for more background.

In mentioned issue, in the Python code is used certifi package which substitute central OS Windows root certificates management.

For me it's still not clear, why it works on some winboxes and on some not.

OS Windows certificates are updated automatically, but may be disabled.

AFAICT there are winGRASS users out there with this issue of not able to install addons.

But there are two problems:

1. Obtain the add-ons list for installation from the server: https://grass.osgeo.org/addons/grass8/modules.xml, and server SSL test is ok.

2. Install add-on, zip package is downloaded from server: http://wingrass.fsv.cvut.cz/ and server SSL test is ok.

First and second server use Let's Encrypt SSL certificate. From October 2021 onwards, only those platforms that trust ISRG Root X1 will validate Let’s Encrypt certificates.

[tmszi]

Is this a GRASS GIS or a Windows problem?

I think that this is a problem with the validation of the SSL Let's Encrypt certificate on the client side (seems central administration of MS Windows root certificates is outdated on some machines, which is used by Python urllib module in GRASS GIS code), but I can't reproduce it.

[hellik]

I've tested it on several win 10 boxes. In some of them this issue occurs, in some of them not.

Could you please check if ISRG Root X1 certificate exists on the computers with this issue?

Result of SSL server test and Let's Encrypt ceritificate compatibility.

1. Launch PowerShell

2. Run command: `Get-Childitem cert:\LocalMachine\root |format-list`

3. See if **ISRG Root X1** certificate exists in the list

yes, it does:

Subject      : CN=ISRG Root X1, O=Internet Security Research Group, C=US
Issuer       : CN=ISRG Root X1, O=Internet Security Research Group, C=US
Thumbprint   : CABD2A79A1076A31F21D253635CB039D4329A5E8
FriendlyName : ISRG Root X1
NotBefore    : 04.06.2015 13:04:38
NotAfter     : 04.06.2035 13:04:38
Extensions   : {System.Security.Cryptography.Oid, System.Security.Cryptography.Oid, System.Security.Cryptography.Oid}

[hellik]

But there are two problems:

1. Obtain the add-ons list for installation from the server: https://grass.osgeo.org/addons/grass8/modules.xml, and  [server SSL test](https://www.ssllabs.com/ssltest/analyze.html?d=grass.osgeo.org&latest) is ok.

2. Install add-on, zip package is downloaded from server: http://wingrass.fsv.cvut.cz/ and [server SSL test](https://www.ssllabs.com/ssltest/analyze.html?d=wingrass.fsv.cvut.cz&latest) is ok.

both fails here.

[hellik]

But there are two problems:

1. Obtain the add-ons list for installation from the server: https://grass.osgeo.org/addons/grass8/modules.xml, and  [server SSL test](https://www.ssllabs.com/ssltest/analyze.html?d=grass.osgeo.org&latest) is ok.

2. Install add-on, zip package is downloaded from server: http://wingrass.fsv.cvut.cz/ and [server SSL test](https://www.ssllabs.com/ssltest/analyze.html?d=wingrass.fsv.cvut.cz&latest) is ok.

both fails here.

g.extension extension=r.stream.distance                                         
Downloading precompiled GRASS Addons <r.stream.distance>...
ERROR: Cannot open URL: http://wingrass.fsv.cvut.cz/grass80/addons/grass-8.0.0/r.stream.distance.zip

though the zip file is there.

not sure, though I guess it is a server side certificate issue.

[tmszi]

not sure, though I guess it is a server side certificate issue.

Could you try some another web server which use Let's Encrypt certificate (e.g. https://www.fsf.org and https://fsfe.org/) from Python shell please?

from six.moves.urllib import request as urlrequest
HEADERS = { "User-Agent": "Mozilla/5.0"}
for url in ["https://www.fsf.org", "https://fsfe.org/"]:
    request = urlrequest.Request(url, headers=HEADERS)
    urlrequest.urlopen(request)

[hellik]

GRASS Version: 8.1.dev                                                          
Code revision: 792b8900d                                                        
Build date: 2022-02-18                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 3.4.1                                                                     
PROJ: 8.2.1                                                                     
GEOS: 3.10.2                                                                    
SQLite: 3.37.2                                                                  
Python: 3.9.5                                                                   
wxPython: 4.1.1                                                                 
Platform: Windows-10-10.0.19044-SP0 

Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
from six.moves.urllib import request as urlrequest
HEADERS = { "User-Agent": "Mozilla/5.0"}
for url in ["https://www.fsf.org", "https://fsfe.org/"]:
    request = urlrequest.Request(url, headers=HEADERS)
    urlrequest.urlopen(request)
    
Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 1253, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 1299, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 1248, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 1008, in _send_output
    self.send(msg)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 948, in send
    self.connect()
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\http\client.py", line 1422, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: A failure in the SSL library occurred (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<input>", line 3, in <module>
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 517, in open
    response = self._open(req, data)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "C:\Program Files\GRASS GIS 8.1\Python39\lib\urllib\request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error A failure in the SSL library occurred (_ssl.c:1129)>

[isaacullah]

Hi all, I am teaching with GRASS this semester, and two of my students, both using WinGRASS, are reporting this same bug. Here is the error message sent to me by one of the students:

Exception in thread
Thread-14
:
Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS
8.0\Python39\lib\threading.py", line 954, in
_bootstrap_inner

self.run()
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\core\gthread.py", line 154, in __run

self.__run_backup()
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\core\gthread.py", line 117, in run

ret = vars()["callable"](*args, **kwds)
  File "C:\Program Files\GRASS GIS
8.0\gui\wxpython\modules\extensions.py", line 369, in Load

raise GException(_("Unable to load extensions. %s") % msg)
core.gcmd
.
GException
:
Unable to load extensions. Download file from
<https://grass.osgeo.org/addons/grass8/modules.xml>, failed.
File is not on the server or check your internet connection.

Tried to find a work around by simply manually installing scripts to GRASS_ADDON_BASE, but for some reason could not execute these files from within GRASS. I am not a Windows user, so likely there is some better way to manually install user scripts in WinGRASS, but this is the way I do it on Linux with no issues?