gocrawl
gocrawl copied to clipboard
Parse <img> tags for crawling, if the option ParseImageTags is set.
Implement the code for #69 Parsing image tags to include them in crawling.
By default the new setting ParseImageTags is set to false, so the current default behavior does not change.
Hello Peter,
Thanks for the PR, makes sense and looks good to me, could you just add a unit test that exercises that new feature (w.opts.ParseImageTags set to true, and checking that image source URLs have actually been added to the crawler)?
Thanks, Martin