Harvest Add support for robots.txt

We should have built in features to handle robots.txt.

Feb 25 '14 07:02 pinscript

Is it still open for contribution?

May 25 '17 12:05 adeelkhalid992

Hi,

It is, but I do not use this project anymore. I'd recommend sjdirect/abot or another crawler that uses the new async HttpClient.

May 26 '17 08:05 pinscript

It would be much appreciated if you just tell me which pattern have you used for handling of threads and also for fetching links from websites, so, It could be easy for me to dig into this project and modify it with latest concepts of c#.

May 31 '17 07:05 adeelkhalid992

Sorry for the late reply, been away.

There is not much to it really. I am using the APM Model (https://msdn.microsoft.com/en-us/library/ms228963(v=vs.110).aspx) in https://github.com/alexandernyquist/Harvest/blob/master/Harvest/Downloader.cs.

Then there are two threads which controls all Downloaders at https://github.com/alexandernyquist/Harvest/blob/master/Harvest/DownloaderQueue.cs.

Links are scraped using the HtmlAgilityPack in the Page-class (https://github.com/alexandernyquist/Harvest/blob/master/Harvest/Page.cs).

But please, do not build on this code. It is way better to use modern techniques using async/await, System.Net.Http.HttpClient.

I can spike out an example for you if you'd like?

Jun 05 '17 05:06 pinscript