Remove URL fragment before storing in `Crawly.Middlewares.UniqueRequest`

[tanguilp]

Usually a fragment leads to the same page.

[oltarasenko]

Yes, it will improve the situation! I will add it to the scope.

[oshosanya]

@Ziinc Is this issue still open? Looking for something to work on. If it is still open, please help with a description of the issue.

[matteoredaelli]

I think you should change the file /lib/crawly/middlewares/unique_request.ex

The fragment could be removed with something like

"http://example.com/faqs#one" |> URI.parse |> Map.put(:fragment, nil) |> URI.to_string

I could submit a pull request about this small change

Regards Matteo