Added post filter to avoid parsing the same posts again after restarting the parsing process
Basically, I had an issue when the parser would try to fetch again the same content instead of skipping the post entirely. So, here's my solution for that.
Hopefully, it will work for you as well
Your filter implementation uses a nested loop (crawledUrls.RemoveAll(x => _ignorePosts.Any(y => y.Id == x.PostId));) which is inefficient. For every single post crawled, the filter will need to iterate every ignored post ID again, and every post downloaded is added to the list of ignored IDs. Have you considered using a HashSet<> instead for the internal implementation?
Oh... I see. That definitely will be better than iterating over and over again. I'll change it