nvda
nvda copied to clipboard
nvaccess
NVDA and add-ons are not update when a http proxy is used by the operating system
I am using the latest stable NVDA release on Win7 in an environment where a http proxy has to be used to access the internet. Unfortunately with this system config no automated updates for NVDA or installed add-ons can be downloaded.
Is this a known issue? Can it be resolved somehow? Can the update mechanism of NVDA be reimplemented to also take care of http proxy settings of the operating system, the default browser e.g., or can the proxy settings be made configurable?
Hi, we need to look into where this is failing (Python 2’s urllib, update check thread, etc.). CC @JCSTeh
This isn't much of a mystery. urllib2 doesn't automatically fetch proxy information from the OS, and NVDA doesn't add it. It's not a technical failure, it's a feature which just doesn't exist. The WinHttpGetIEProxyConfigForCurrentUser function can be used to get the information. Other decisions should also be made, such as:
- What sort of proxies should NVDA support? Just HTTP? HTTp and SOX5?
- In addition to fetching the proxy config from the OS, should a panel be added to the NVDA settings dialog to turn this off or enter proxy information for NVDA specifically?
Should we also advise add-on authors to use any configured proxy when making HTTP requests? Or provide a preconfigured HTTP client of some sort that add-ons can import and use, knowing that the correct network config will be in place?
I could swear this has been discussed before (or at least I looked into it), but I can't find an issue for it, nor a mailing list thread.
Actually, both urllib and urllib2 do fetch proxy information from the OS. However, I vaguely recall there's some difference in functionality. Perhaps urllib didn't pass http connections through an https proxy or something like that. Either way, keep in mind that NVDA currently uses urllib, not urllib2. It's worth testing with urllib2 to see if it addresses the problem.
I also vaguely recall investigating switching to urllib2 for some reason,
but again, I can't quite remember the details, nor can I find them.
Hi, wasn’t this discussed in 2014 or 2015 around the time we had to go through SSL changes? Thanks.
hi. @jcsteh you're right, we discussed this before https://github.com/nvaccess/nvda/issues/7893 we need to mark one of those as duplicate. Thanks.
Hello,
I would like to take this issue.
I have no such configuration here but by using fiddler I have been able to emulate what I believe to be the same situation. I can take a look at it and see if I can fix it.
Can I go ahead?
Because I created this issue, I'd be glad to help if I can. If you have no environment with an http proxy, I can test it in my companies network, thats where I have the described problem.
Ok.
I have a working copy with the fix.
The problem is really with the urllib library. Interestingly it is discovering proxies correctly on Windows (which is the only OS we are interested in) but the urlopen method is failing even when the proxies resolved by urllib itself are issued in the proxies parameter.
For reference, the urlib.getproxies() method returned no proxy when I was behind a proxy and the correct list of proxies (one for http and one for https) when I had fiddler set up as a system proxy.
If this is a bug on urllib or not I don't know. There was no sense to investigate because python 2.7 and 3 already have a new library, called urllib2.
In its documentation, it is specified that the urlopen method of urllib2 auto discovers proxies and setup the connection accordingly. My tests confirm this, as importing the newer library made the process work seamlessly with and without a proxy.
Will this solve all possible problems we might ever have with proxies? Hardly. There might be proxies that urllib2 can not handle, the users operating system might or might not be configured to detect proxies automatically and so and so ... but this will possibly solve the vast majority of problems encountered inn the most common scenarios.
@sonor3000 I will try to generate a working copy of NVDA and make it available for you to test.
Be aware that this copy will pretend it is a release version of NVDA but it is not. It will pretend it is a release version because test versions do not offer the update option in the help menu and thus it would be impossible to test the feature.
Be also aware that I can not be held responsible for any problems caused by the run of this copy and be aware that this is a strictly based test version that should be deleted as soon as your tests finish.
If you can contact me via e-mail I can provide you and anyone else wanting to test the thing with a link.
If tests in real environments are successful I will submit the pr.
On Sun, November 25, 2018 11:01 pm, Marlon Brandão de Sousa wrote:
I have a working copy with the fix.
Cool :-).
@sonor3000 I will try to generate a working copy of NVDA and make it available for you to test.
[...]
If you can contact me via e-mail I can provide you and anyone else wanting to test the thing with a link.
Sorry, I do not have your mailaddress, but you can contact me via [email protected].
I can test the fix, but on the other hand if nothing is downloaded because no update is offered, its hart to see if things are working or not. Maybe I can download a older version of a plugin, install it and see if it is updatted...
All the best and thanks for your work and help,
Schoepp
Hello,
Here is what I carefully have found after some investigation.
I ask @josephsl and @leonardder and who else you think is needed to read this carefully as it will be the bases for a proposal soon to be written.
To begin wit I must say that portability between operating systems are not being taken in account, since NVDA is and will for a foreseeable future be a project running only on Windows.
When using fiddler as a proxy, urllib.getproxies method consistently failed to detect I was behind a proxy. Even with me issuing the correct proxy address in the proxy parameter of urlopen the connection was not being established. When using urllib2, the urlopen method was able to auto discover that I was behind a system proxy and could establish connections correctly. I then took the copy of NVDA using urllib2 and tried it behind a real corporate proxy. This one has a .pac file set up via an administrative script. This time, urllib2.urlopen failed to detect the proxy and the connection was not established. Python 3 urllib implementation is based on python2 urllib2 implementation, so I have good reasons to believe that its algorithm is not prepared to auto discover given kinds of proxy setup on Windows. By researching, it was clear that python has no built-in, universal method to consistently discover and deal with proxies on Windows. This is the reason why the pypac project https://github.com/carsonyl/pypac was created.
So, it was very clear for me that if we really want to make NVDA HTTP connections more globally available, including for machines behind multiple kinds of proxies we would have to change the current implementation strategy. The pipac project is great, but I started to think that including this dependency in NVDA was a little over kill. The mean reasons are:
1- The urllib and urllib2 libraries of python 2 and the urllib library of python 3 are low level pieces of code that are making HTTP connections available for python on multiple operating systems. However, because of this interoperability, it is clear that those libraries are not dealing with more specific maters, proxy being one of them. 2- Pypac would bring yet another dependency, making the project somewhat harder to maintain. 3- but, in the other hand, Windows is clearly able to resolve its own HTTP questions, including proxies by its on. 4- And Microsoft offers HTTP services for its client applications via com automation. 5- NVDA Will run only on Windows, so nothing offering portability is really an advantage in this case. In the other hand, using the high level windows services will separate the concerns of NVDA from the concerns of HTTP connectivity, making the code cleaner, providing correct connectivity management and making updates on connectivity and security maters bound to the current operating system.
With this in mind, there was no reason, at least in my perspective, to keep trying to deal with urllib or with external packages. I needed to check what services were being exposed and how those services were exposed.
By investigating more, I have found two candidates: 1- Microsoft xml (https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ms759148(v%3Dvs.85). 2- WinHTTPRequest (https://docs.microsoft.com/en-us/windows/desktop/winhttp/winhttprequest).
I have chosen winHTTPRequest. Reasons are several, the main one being that msxml has several versions and some of them tend to be made available with more modern versions of office. The guids of the com object change for new versions. In the other hand, WinHTTPRequest 5.1 is supported since xp to 10 and nothing points towards it being unsupported. In fact, if you search for things such as proxy and Windows, WinHTTPRequest will often be listed as the solution adopted. The comtypes library was queen on helping me to test a quick prototype without even needing to generate yet another nvda copy for testing. In fact, when I was behind that corporate proxy, issuing those lines validated the solution:
url = "https://www.google.com" from comtypes import client client.gen_dir = None # needed otherwise contypes fail when using the console httpClient = client.CreateObject("WinHTTP.WinHTTPRequest.5.1") httpClient.open("get", url) httpClient.send()
After that, typing httpClient.status
printed 200
and typing
httpClient.ResponseText
printed a nice HTML with google's page.
I then went home and coded the updater to use WinHTTPRequest instead of urllib. Another dist copy of NVDA was generated and taken to the machine behind the proxy and everything worked great.
I am also almost certain that the use of an operating system HTTP service will solve #4803 but I am not sure.
What next?
I am fully convinced that this is by far the best strategy we can use. It separates concerns, it makes NVDA code more clear, it adds no hurdles and it guarantees that Microsoft new requirements and complex situations on HTTP will be solved automatically, all with a cost of being tied to a specific operating system implementation that does not exist at all because NVDA itself is tied to Windows anyways.
Because of that, I will create a tiny wrapper to WinHTTPRequest that will provide NVDA code with a high level api to: 1- Make HTTP connections. 2- Get results synchronously or asynchronously. 3- provide specific call backs with partial data if the connection is asynchronous.
This is needed so that neither NVDA modules nor other addons get their code cluttered with http specific maters and also to provide a service with logging and other facilities added to make connectivity easier.
Finally, one last benefit is that urllib, urlib2 of python 2 and urllib of python 3 present syntactical differences, which make the code implementing the use of all these libraries at the same time to support python 2 and python 3 a really mess dealing with import as and handling import errors ...
Well, I can provide a branch with all this in place if there are no objections.
Independently of all this, I will send @sonor3000 a link with the WinHTTPRequest implementation to see if he / she can also at least get the update dialog to work.
Thanks, Marlon
hi. since I too am behind a broxy at work, and I have reported this issue initially, I can certainly test your copy if that helps. my email address is: [email protected]
Hi Marlon,
On Sat, 01 Dec 2018 15:39:48 I can provide a branch with all this in place if there are no
objections.
Independently of all this, I will send @sonor3000 https://github.com/sonor3000 a link with the WinHTTPRequest implementation to see if he / she can also at least get the update dialog to work.
Now, since NVDA 2018.4 is out, I can test if the update is working with the proxy configured in my working environment. I'll block the update till I get a version with your changes that I can test.
Cheers and thanks for all your work,
Chris
Hello,
First of all I am sorri for the delay .. comtypes may Be a very whild, unexplored and aggressive land when you want to do more than simple calls to objects methods. With this out of the way I need you perform a simple test with your current copy: activate nvdda menu and then help and search for updates.It is expected that after a time a message reporting error when searching for updates occurs. Can you confirm that this is true in your case? If so, my copy will search successfully for updates, but won’t download it at this time. This will confirm that my strategy for dealing with proxies is working the way it should. If so, then I can build the new updater with confidence. In my case, it worked. If it also does in your keys, I’ll be probably finishing my work in two or three weeks and submit it for the NV access guys. I will be sending my copy to you in probably one or two days, but before, I would like that you confirm the info I’ve just requested.
Obrigado, Marlon
Em 17 de dez de 2018, à(s) 13:16, sonor3000 [email protected] escreveu:
Hi Marlon,
On Sat, 01 Dec 2018 15:39:48 I can provide a branch with all this in place if there are no
objections.
Independently of all this, I will send @sonor3000 https://github.com/sonor3000 a link with the WinHTTPRequest implementation to see if he / she can also at least get the update dialog to work.
Now, since NVDA 2018.4 is out, I can test if the update is working with the proxy configured in my working environment. I'll block the update till I get a version with your changes that I can test.
Cheers and thanks for all your work,
Chris
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
Bon dia Marlon,
On Mon, December 17, 2018 8:08 pm, Marlon Brandão de Sousa wrote:
First of all I am sorri for the delay ..
No problem, don't stress :-).
activate nvdda menu and then help and search for updates.It is expected that after a time a message reporting error when searching for updates occurs. Can you confirm that this is true in your case?
Yes, I can confirm this. The message, that an error occured while searching for updates, is displayed right after activating the regarding menu entry.
I will be sending my copy to you in probably one or two days, but before, I would like that you confirm the info I’ve just requested.
Confirmed :-). If you send me a copy of your work and tell me what I need to do and how to test, I'd be glad to help.
Cheers from Munich,
Chris
Hello @sonor3000 and @mohdshara ,
I have just sent you both an e-mail with link and further instructions.
I generated a dist copy of NVDA from a branch somewhat outdated, little after 2018.3, and implemented http connection via winHTTPRequests in the update checking process. As a consequence, now at least the update checking process, leading to a dialog asking you to update, should show whereas the same dialog, activated when going to NVDA menu then help then search for updates should present an error in the official copies of NVDA that still use urlib to manage HTTP connections.
Should the tests succeed on this patched version, I can code it into the main NVDA repo and open a pull request.
Please do replay my e-mail reporting if testing is successful and also update this issue here with your findings so everyone is tracked.
Please do not use this copy for anything else than testing.
Please do not try to update this copy (e.e do not reply yes in the dialog when asked to update) because the downloader is not yet with WinHTTPRequest, it is using urllib. The goal here is not to test the update process but is ratter to check the new WinHTTPRequest implementation which is, again, only implemented in the update checking process.
Please do delete this copy as soon as tests are performed. There is nothing worth in this version for day to day usage.
Thanks, Marlon
Hi Marlon,
On Fri, December 21, 2018 3:35 am, Marlon Brandão de Sousa wrote:
I have just sent you both an e-mail with link and further instructions.
Thanks for the file...
Unfortunatly bove tests you want us to try did not work for me or my proxy setup :_(.
- While staarting NVDA there was no message that a newer release is available.
- While searching for upgrades via the help menu only the usual error appeared, like in the normal NVDA releases.
Just let me know if I can do more tests or if you need more information about my setup or whatever.
Cheers and thanks for your work,
Chris
@marlon-sousa Could be I missed this, but... have you actually tested all scenarios with urllib on Python 3? I recall you only mention urllib and urllib2 on python 2. As we have threshold_py3_staging snapshots, it should be possible to test NVDA update checking without too much hassle.
@marlon-sousa could you please give an update on this? See last comment posted by @leonardder above.
Hello,
Sorry for the response delay.
I have left the office where I had similar conditions in terms of proxy to test and have no longer how to do it.
Trials to simulate the exact proxy used and settings out of that office failed.
My intention was to use Microsoft http client via com plus. At that time, I could bypass my own simulated proxy but trial builds still failed on @sonor3000 scenarios.
Other than that, we are now on python 3, using urllib 3 and may be things changed.
It would be nice to know from @sonor3000 if this is the case after 2019.3.
Hi Marlon,
On Sun, Nov 08, 2020 at 04:53:41AM -0800, Marlon Brandão de Sousa wrote:
Other than that, we are now on python 3, using urllib 3 and may be things changed.
Sorry, I can't tell :-(. Also my working environment changed and I ahve a new system and a totaly different proxyconfiguration now.
What I can say is, that I get notified if a new NVDA release is available.
But downloading the new release is not possible, because .exe files are blacklisted in general. So I have to let whitelist the URL and start the download and do an manual installation to get a new version running, I can not longer use the normal update mechanissm in my working environment :-(.
So, maybe the issue has to be closed, because no further testing is possible....
Cheers,
Schoepp
