frontera
frontera copied to clipboard
Handling encoding of URL's.
This problem arises when the URL is non-ascii. We can either store the URL's encoding but this introduces performance deterioration. Another option is to convert the URL's to an ascii representation such as - safe_url_string which is also used by scrapy, however this function also affects performance.
It would be good to improve performance on that safe_url_string and use it on Request instantiation, probably.