FTP concurrent requests problem

Open draganpodvezanec opened this issue 7 years ago • 1 comments

Hi.

I have put

CONCURRENT_REQUESTS = 1 CONCURRENT_REQUESTS_PER_IP = 1 CONCURRENT_REQUESTS_PER_DOMAIN = 1 CONCURRENT_ITEMS = 1

in ftptree_crawler/settings.py

Anyway, soon after starting crawler, I get an error:

ConnectionLost: ('FTP connection lost', <twisted.python.failure.Failure twisted.protocols.ftp.CommandFailed: ['530 Sorry, the maximum number of clients (6) from your host are already connected.']>)

FTP server is limited to 6 concurrent connections.

Does FTPTREE obey these CONCURRENT settings while crawling?

Feb 14 '18 23:02 draganpodvezanec

Thanks for submitting the issue. I don't have time to maintain this, though. Feel free to submit a PR if you figure it out.

Feb 23 '18 23:02 laserson