Replace HttpWebRequest with HttpClient

[rogerfar]

trafficstars

I came across this issue recently: https://stackoverflow.com/questions/66156483/webrequest-behavour-different-between-net-framework-4-8-and-dotnet-5 and https://twitter.com/mikaelsyska/status/1359968961765535744

It might be worth looking into replacing the HttpWebRequest in ChunkDownloader with something like a HttpClient, which has been completely rewritten in .NET Core and should yield better performance.

[ghost1372]

similar to https://github.com/bezzad/Downloader/issues/31 And yes, I agree with that😁

[bezzad]

I'm researching and I hope I can find a solution soon. Because the structure cost of the program for conversion and full use of HttpClient is expensive.

[hez2010]

I think HttpWebRequest can be replaced with HttpRequestMessage, the later one is used by HttpClient and almost the same with HttpWebRequest. Also note that HttpClient should be used as singleton (it won't release socket even after calling Dispose()).

[GihanSoft]

There is a problem with http client. Buffer of HttpClient is limited to 2GB. Won't it make problems?

[hez2010]

@GihanSoft You can buffer it by yourself while using HttpClient:

var req = new HttpRequestMessage(...);
var res = (await httpClient.SendAsync(req, HttpCompletionOption.ResponseHeadersRead)).EnsureSuccessStatusCode();
using var stream = await res.Content.ReadAsStreamAsync();
const int bufferSize = 1048576;
var buffer = new byte[bufferSize];
while (true)
{
    var length = await stream.ReadAsync(buffer);
    if (length == 0) break;
    // use buffer[0..length]
}

[GihanSoft]

@hez2010 The point is you can't control flow of date. no way to pause it or slow it. If you set low buffer capacity, it won't wait for buffer to be read. It just throw.