I got ban from TWSE after running the unit test twice

[ianlini]

I got ban from TWSE after running the unit test twice. I am very sure that I am not running other crawler at the same time. The unit test is very dangerous. Maybe we should add some delay between requests, or mock almost all the requests.

[mlouielu]

What kind of ban do you get? The whole RealtimeTest get failed? I'm currently fixing this issue for the unittest.

[ianlini]

While calling TWSEFetcher.fetch, I got [Errno 101] Network is unreachable, and my browser cannot open http://www.tse.com.tw/. It's a typical ban when you request TWSE too frequently.

[mlouielu]

I can reproduce this when doing the unittest twice

...
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.twse.com.tw', port=80): Max retries exceeded with url: /exchangeReport/STOCK_DAY?date=20180101&stockNo=2330 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5432992898>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

[Evanstsai]

我也有類似的問題 我成功執行一次，但第二次就不work了 HTTPConnectionPool(host='www.twse.com.tw', port=80): Max retries exceeded with url: /exchangeReport/STOCK_DAY?date=20180401&stockNo=2303 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10f8c40f0>: Failed to establish a new connection: [Errno 61] Connection refused',))

[mlouielu]

I've try using request.Session, but it still don't work :(

someone said the limit is 5s/3 query