dlang-requests
dlang-requests copied to clipboard
ikod
Windows proxies/certificates using WinHTTP
Maybe dlang-requests should additionally wrap WinHTTP on Windows for not needing any dependencies like SSL.
It can take the system proxy (if http_proxy is not set) and has a lot of additional internal capabilities like kerberos authentication which I think would be pretty cool to have. Also it's recognized for example by anti-virus tools and other utilities to have all HTTP requests tracked using a central API. I think there could be a good wrapper around it in the library.
It would be cool if it would be compatible with vibe.d tasks using the async methods which integrate in the rest of Win32 events.
https://docs.microsoft.com/en-us/windows/desktop/WinHttp/about-winhttp
I've not been able to get the OpenSSL stuff working with this library, so I'm going to fork it and give a WinHTTP wrapper a shot. Shouldn't be too hard (famous last words).
I wrote some WinHTTP code to download a file if you want to look at it for reference: (though there might be better ways, it's a way that worked and is stitched together from the examples and documentation) https://github.com/Pure-D/serve-d/blob/master/http/source/served/http.d#L38
I've already got working code using WinHTTP in D by copy-pasting MS example code, D-ifying it, and then lots of trial and error :).
Your example looks like Wininet though. Well, wininet will let me handle FTP as well. I'll stick to WinHTTP for the HTTP side as I've already figured out how to do headers and such on that.
Hello, Is is possible to adapt windows ssl library (SecureChannel?) to solve this problem? It would be nice to add this functionality. I'm very weak with windows programming, so I afraid to start this job.
So for WinHTTP SSL is handled very easily. The function WinHttpOpenRequest is called with an SSL flag, like so: WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE);
That WINHTTP_FLAG_SECURE is literally it, assuming you gave the correct port in the earlier call to WinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works.
So for WinHTTP SSL is handled very easily. The function
WinHttpOpenRequestis called with an SSL flag, like so:WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE);That
WINHTTP_FLAG_SECUREis literally it, assuming you gave the correct port in the earlier call toWinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works.
But with WinHTTP we will lose some dlang-requests functionality like receiveAsRange or not? If not - I'm ok with WinHTTP for windows builds.
ReceiveAsRange is actually kind of the default of how WinHTTP handles response data—you read it in chunks. My example code stitched these chunks together with ~= on a dynamic array, but we could easily make a wrapper to return the original chunks instead.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10
From: ikod [email protected] Sent: Saturday, July 27, 2019 10:03:52 AM To: ikod/dlang-requests [email protected] Cc: TransientResponse [email protected]; Comment [email protected] Subject: Re: [ikod/dlang-requests] Windows proxies/certificates using WinHTTP (#97)
So for WinHTTP SSL is handled very easily. The function WinHttpOpenRequest is called with an SSL flag, like so: WinHttpOpenRequest(hConnect, "METHOD"w.toUTF16z, "/route"w.toUTF16z, null, WINHTTP_NO_REFERER, WINHTTP_DEFAULT_ACCEPT_TYPES, WINHTTP_FLAG_SECURE);
That WINHTTP_FLAG_SECURE is literally it, assuming you gave the correct port in the earlier call to WinHttpConnect. Windows does the whole SSL connection thing itself. Totally transparent from the point of view of the API. For non-HTTPS calls, passing a flag value of 0 works.
But with WinHTTP we will lose some dlang-requests functionality like receiveAsRange or not? If not - I'm ok with WinHTTP for windows builds.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fikod%2Fdlang-requests%2Fissues%2F97%3Femail_source%3Dnotifications%26email_token%3DABJLBRAI5MJGK6PY2JFUHKLQBRIURA5CNFSM4HCADZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26MBJQ%23issuecomment-515686566&data=02%7C01%7C%7C342479bf62084bc88a5608d7129b4225%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998330340953219&sdata=0%2FshOGRVvCkXoCixudWCYE58huhykRm%2F%2BekUXmzmr4k%3D&reserved=0, or mute the threadhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABJLBRAB2YNK5XDIBS6GEVDQBRIURANCNFSM4HCADZDA&data=02%7C01%7C%7C342479bf62084bc88a5608d7129b4225%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998330340963224&sdata=8QwxytNhcF%2FCaTE1REYcagCAWTN4SxVrA8H8RcNu3bs%3D&reserved=0.
Thanks @TransientResponse !
Then I think it is worth to try. Did you forked this repo in order to implement this feature?
It’s forked here: https://github.com/TransientResponse/dlang-requests
First small, non-functional commit coming shortly. I’m heading out of the house soon.
Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10
From: ikod [email protected] Sent: Saturday, July 27, 2019 10:18:23 AM To: ikod/dlang-requests [email protected] Cc: TransientResponse [email protected]; Mention [email protected] Subject: Re: [ikod/dlang-requests] Windows proxies/certificates using WinHTTP (#97)
Then I think it is worth to try. Did you forked this repo in order to implement this feature?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fikod%2Fdlang-requests%2Fissues%2F97%3Femail_source%3Dnotifications%26email_token%3DABJLBRH36YSXPPJREM6NWD3QBRKK7A5CNFSM4HCADZDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26MI4Q%23issuecomment-515687538&data=02%7C01%7C%7Ce0ab41ce3a21472ab49908d7129d492c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998339049309773&sdata=6uJbfu%2B3lx4LkQqMpuQi5Agd66IuwVbIRGIwxQ0Z3Fo%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABJLBRA5TQMTHJBVGDHIQ7TQBRKK7ANCNFSM4HCADZDA&data=02%7C01%7C%7Ce0ab41ce3a21472ab49908d7129d492c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636998339049319778&sdata=4alyzw4C49Pa4ubryCMjujhvGM8pE7o776WjbAwOlsE%3D&reserved=0.
Well, my first implementation works and 11/12 unittests pass (unless dub is tricking me). The test that's failing is package.d:80:__unittest_L29_C9 Check handling incomplete status line, and the reason seems to be that a GET to /incomplete on httpbin.org (local or remote) is returning 404 and not 600 or 502 as the assert wants. Not sure why this is.
Well, my first implementation works and 11/12 unittests pass (unless dub is tricking me). The test that's failing is
package.d:80:__unittest_L29_C9 Check handling incomplete status line, and the reason seems to be that aGETto/incompleteon httpbin.org (local or remote) is returning 404 and not 600 or 502 as the assert wants. Not sure why this is.
Very good!
You can enable full debug on particularly this test with globalLogLevel = LogLevel.trace and Request.verbosity = 2 or 3; You can place debug output here, I'll check
I'm so sorry for the long delay on this, life got hectic for me at that time and I got pulled away by other things. I'll try to get back into it tomorrow and update you then.
On Friday, November 12, 2021 6:40 AM (-05:00), ikod wrote:
hello @TransientResponse
any news on this?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.
-- Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com
I set loglevel to trace and verbosity on the failing request to 2, and I only get a little more info:
2021-11-13T08:57:35.574 [info] source\requests\package.d:82:__unittest_L29_C9 Check handling incomplete status line
2021-11-13T08:57:35.575 [trace] source\requests\http.d:2616:execute serving Request(GET, http://127.0.0.1:8081/incomplete)
Doing connection to 127.0.0.1 with scheme http
Doing GET for /incomplete
2021-11-13T08:57:35.580 [trace] source\requests\streams.d:1000:connect Create connection to 127.0.0.1:8081
2021-11-13T08:57:35.580 [trace] source\requests\streams.d:1009:connect Trying 127.0.0.1:8081
2021-11-13T08:57:35.581 [trace] source\requests\streams.d:1019:connect Connected to 127.0.0.1:8081
2021-11-13T08:57:35.581 [trace] source\requests\streams.d:981:close Close socket
core.exception.AssertError@source\requests\package.d(86): unittest failure
The trace lines from streams.d look as though it's not calling the WinHttp API at all in that test, but I know it is compiling and testing. I can get errors out of WinAPI if I don't have an httpbin server running locally (like in docker). Unless the unittest build overrides version(Windows) checks?