Got banned after 4 requests (3 sec gap)

[shinkuan]

每次測試都在4次請求後被禁, 請求與請求之間有三秒間隔, 以下為代碼

`import twstock import time

// Create a Stock array stock = []

// stock[i] is stock which sid = i for i in range(0, 10000): sid = str(i) # check if stock i exist if sid in twstock.twse or sid in twstock.tpex: stock.append(twstock.Stock(sid)) # print(stock[i].data) print('sid' + stock[i].sid + 'get!') time.sleep(3) # if not, its NaN else: stock.append('NaN') print('stock' + sid + 'does not exist')`

以下為輸出結果:

stock0does not exist stock1does not exist stock2does not exist stock3does not exist . . . stock1097does not exist stock1098does not exist stock1099does not exist stock1100does not exist sid1101get! sid1102get! sid1103get! sid1104get! stock1105does not exist stock1106does not exist stock1107does not exist Traceback (most recent call last): File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse response.begin() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin version, status, reason = self._read_status() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\adapters.py", line 439, in send resp = conn.urlopen( File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen retries = retries.increment( File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\util\retry.py", line 403, in increment raise six.reraise(type(error), error, _stacktrace) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\packages\six.py", line 734, in reraise raise value.with_traceback(tb) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 426, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\urllib3\connectionpool.py", line 421, in _make_request httplib_response = conn.getresponse() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 1332, in getresponse response.begin() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 303, in begin version, status, reason = self._read_status() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\http\client.py", line 272, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:/Users/USER/PycharmProjects/StockTrade/app.py", line 11, in stock.append(twstock.Stock(sid)) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 154, in init self.fetch_31() File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 183, in fetch_31 self.fetch_from(before.year, before.month) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 175, in fetch_from self.raw_data.append(self.fetcher.fetch(year, month, self.sid)) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\twstock\stock.py", line 58, in fetch r = requests.get(self.REPORT_URL, params=params, File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\api.py", line 76, in get return request('get', url, params=params, **kwargs) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "C:\Users\USER\PycharmProjects\StockTrade\venv\lib\site-packages\requests\adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Process finished with exit code 1

有試過用更長時間 (10 sec) 為間隔 依然被禁

[joytsay]

我發現可以隨機給予3-10秒sleep (抓取中華電信20年成功):

in stock.py (line 55)

    def fetch(self, year: int, month: int, sid: str, retry: int=5):
        params = {'date': '%d%02d01' % (year, month), 'stockNo': sid}
        for retry_i in range(retry):
            randNum = random.randrange(3,10)    ##增加這兩行##
            time.sleep(randNum)                 ##增加這兩行##
            r = requests.get(self.REPORT_URL, params=params,
                             proxies=get_proxies())

[rach-lyn]

請問這裡的self是要填什麼啊？謝謝> 我發現可以隨機給予3-10秒sleep (抓取中華電信20年成功):

in stock.py (line 55)

    def fetch(self, year: int, month: int, sid: str, retry: int=5):

        params = {'date': '%d%02d01' % (year, month), 'stockNo': sid}

        for retry_i in range(retry):

            randNum = random.randrange(3,10)    ##增加這兩行##

            time.sleep(randNum)                 ##增加這兩行##

            r = requests.get(self.REPORT_URL, params=params,

                             proxies=get_proxies())

[joytsay]

self 只是 python 語言裡面 class 的 instance 這裡為了要避掉 class TWSEFetcher(BaseFetcher)中 fetch fucntion 抓太快被台股 server 鎖IP的問題 在twstock/stock.py#L57 的下一行 for 迴圈裏面 增加 以下兩行 code 即可

randNum = random.randrange(3,10)   
time.sleep(randNum) 

呼叫方法可以參考我的Jupyter Notebook stock_from = stock.fetch_from(2000,10) # 獲取 2010 年 01 月至今日之股票資料