pyspider icon indicating copy to clipboard operation
pyspider copied to clipboard

Error in scheduler

Open wax8280 opened this issue 6 years ago • 1 comments

  • pyspider version:0.3.10
  • Operating system:Deepin os
  • Start up command:

Hello.I run pyspider with redis&mysql.At first,it works.But a few minutes later,it doesn't work.It's seems that there is an error in shceduler.

Fatal Python error: GC object already tracked

Thread 0x00007f04f67fc700 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/tornado/ioloop.py", line 863 in start
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 783 in xmlrpc_run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f04f6ffd700 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 392 in cmd_query
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 264 in execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 37 in _execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 74 in _select2dic
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/mysql/taskdb.py", line 95 in get_task
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 791 in on_request
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1221 in _thread_worker
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f04f77fe700 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 392 in cmd_query
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 264 in execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 37 in _execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 74 in _select2dic
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/mysql/taskdb.py", line 95 in get_task
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 791 in on_request
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1221 in _thread_worker
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f04f7fff700 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 392 in cmd_query
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 264 in execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 37 in _execute
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 120 in _update
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/mysql/taskdb.py", line 137 in update
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 322 in update_task
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 846 in on_old_request
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 793 in on_request
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1221 in _thread_worker
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007f04fcb38700 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 348 in fetch_eof_columns
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 288 in get_rows
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 303 in get_row
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/mysql/connector/cursor_cext.py", line 532 in fetchone
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/basedb.py", line 77 in _select2dic
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/database/mysql/taskdb.py", line 95 in get_task
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 791 in on_request
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1221 in _thread_worker
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 864 in run
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/lib/python3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007f0505b1d080 (most recent call first):
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1253 in _wait_thread
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 1272 in run_once
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/scheduler/scheduler.py", line 652 in run
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/run.py", line 213 in scheduler
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/decorators.py", line 17 in new_func
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/core.py", line 555 in invoke
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/core.py", line 956 in invoke
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/core.py", line 1137 in invoke
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/core.py", line 717 in main
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/click/core.py", line 764 in __call__
  File "/usr/local/lib/python3.6/lib/python3.6/site-packages/pyspider/run.py", line 754 in main
  File "/usr/local/lib/python3.6/bin/pyspider", line 11 in <module>
[W 181227 19:34:01 index:108] connect to scheduler rpc error: ConnectionRefusedError(111, 'Connection refused')

Thanks for your answer.

wax8280 avatar Dec 27 '18 11:12 wax8280

i have repeat this error, when i update the mysql-connector-python package, if the package up 8.0.12, will occur the error, you can try pip install mysql-connector-python==8.0.12

ider-zh avatar Apr 03 '19 03:04 ider-zh