Mikhail Korobov

I'm late to the party, but what about renaming it to UrlAuthMiddleware? 1. ~~This middleware is not handling other HTTP Basic and FTP auth sources like spider attributes, request.meta keys,...

+1 to merge once https://github.com/scrapy/scrapy/pull/1466#issuecomment-281724763 is addressed

@umrashrf yeah, I meant FormRequest or something similar to https://github.com/TeamHG-Memex/autologin-middleware, and https://github.com/scrapy/scrapy/issues/40. But don't worry about this comment, I don't have a better name anyways :)

@umrashrf my comment was only about names. autologin-middleware will work with AuthMiddleware. `AuthMiddleware` handles auth, but it doesn't handle all kinds of auth, only a couple of specific auth cases....

@redapple I haven't though about this option (use two middlewares), but now as you wrote it this sounds like a good idea, I like it.

@umrashrf this name makes sense, but I also like @redapple's suggestion to create two middlewares: they have almost nothing in common.

By the way, this also fails: ``` yield scrapy.Request("http://scrapy.org", self.parse) ``` _-_ the request is filtered out because of https://github.com/scrapy/scrapy/issues/1225 and self.parse is never called. Crawling scrapy.org with Scrapy is...

The most evil spider ever: looks innocent, but doesn't work for multiple reasons ``` py import scrapy class ScrapySpider(scrapy.Spider): name = 'scrapyspider' def start_requests(self): yield scrapy.Request("http://scrapy.org", self.parse_main) def parse_main(self, response):...

@jlbren yeah, it is still open; help is appreciated!

Thanks @jlbren! Feel free to ask any questions and discuss it here. The issue doesn't look easy because it is a bug in urlparse.urljoin from standard library. Likely we'll have...