abot
abot copied to clipboard
sjdirect
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
The `PageRequester.MakeRequestAsync(...)` method [calls](https://github.com/sjdirect/abot/blob/ccd9f6dc98fe1292a892519d6eadb7bdc1322dde/Abot2/Core/PageRequester.cs#L74) the `_httpClient.SendAsync(...)` method without specifying the `HttpCompletionOption` parameter. The default behavior is to download the content.
The crawl of a site supporting only TLS 1.2 was failing. Modifying the constructors of the PageRequester assigning the default security protocol resolves the problem for us. This may not...
I believe it is the same issue as https://github.com/sjdirect/abot/issues/206 which was closed based on "integration unittest is passing". This UT is passing: ``` [Test] public async Task Crawl_Synchronous_CancellationTokenCancelled_StopsCrawl() { var...
In dot net, on Linux (not in Windows), the `Uri.TryCreate` method with the `UriKind.Absolute` parameter returns true if the URL is relative.
I am trying to crawl this page https://www.tzb-info.cz/kontakty By passing it to validUri in the following code: ``` var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor()); var crawledPage = await...
I want to force the ssl protcols to 1.2 as Windows 11 defaults to 1.3 and my server does'nt support 1.3 The configuration extensions dictionary only seems to be store...
Hi there! Your fantastic library is spamming my application logs 😆 Would you **accept a PR** that adds `.ForContext` where applicable so that the log levels for your namespace can...
