stratum-mining-proxy icon indicating copy to clipboard operation
stratum-mining-proxy copied to clipboard

Published 20 hours ago • verified publisher icon slush0

Extranonce slots are full, please disconnect some miners

Open alphaonex86 opened this issue 11 years ago • 32 comments

Hello, I have: 2013-04-23 15:51:26,901 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:27,149 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:27,602 INFO proxy stratum_listener.submit # [15ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:28,067 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:29,751 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:29,840 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:30,085 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:31,243 INFO proxy stratum_listener.submit # [15ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:32,745 INFO proxy getwork_listener._on_authorized # Worker 'pool' asks for new work 2013-04-23 15:51:33,539 INFO proxy getwork_listener._on_authorized # Worker 'pool' asks for new work 2013-04-23 15:51:35,481 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:38,814 INFO proxy stratum_listener.submit # [14ms] Share from 'pool' REJECTED: (21, u'Stale work', None) 2013-04-23 15:51:41,580 INFO proxy jobs.submit # Submitting d6642a20 2013-04-23 15:51:41,595 WARNING proxy getwork_listener._on_submit # [14ms] Share from 'pool' accepted, diff 1 2013-04-23 15:51:43,635 INFO proxy client_service.handle_event # New job 13d8-1a8 for prevhash a8e7003d, clean_jobs=False 2013-04-23 15:51:44,987 INFO proxy stratum_listener.submit # [14ms] Share from 'alphaonex86_ultracopier' accepted, diff 1 2013-04-23 15:51:47,896 ERROR protocol protocol.process_failure # [Failure instance: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:81:call /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:137:maybeDeferred /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:78:_run /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1213:unwindGenerator --- --- /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1070:_inlineCallbacks /home/stratum-mining-proxy/mining_libs/stratum_listener.py:138:subscribe /home/stratum-mining-proxy/mining_libs/stratum_listener.py:108:_get_unused_tail ] None 2013-04-23 15:51:47,934 INFO stats stats.print_stats # 59 peers connected, state changed 3 times 2013-04-23 15:51:47,983 ERROR protocol protocol.process_failure # [Failure instance: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:81:call /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:137:maybeDeferred /usr/lib64/python2.7/site-packages/stratum-0.2.12-py2.7.egg/stratum/services.py:78:_run /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1213:unwindGenerator --- --- /usr/lib64/python2.7/site-packages/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py:1070:_inlineCallbacks /home/stratum-mining-proxy/mining_libs/stratum_listener.py:138:subscribe /home/stratum-mining-proxy/mining_libs/stratum_listener.py:108:_get_unused_tail How fix? Cheer,

alphaonex86 avatar Apr 23 '13 13:04 alphaonex86

Can you increase the max connected client to 32Bits? Cheer,

alphaonex86 avatar Apr 23 '13 14:04 alphaonex86

Do you have more than 250 clients connected to Stratum proxy?

slush0 avatar Apr 23 '13 15:04 slush0

In reallity more than 20000 try use my proxy just now, I target 100 000 into few months.

alphaonex86 avatar Apr 23 '13 15:04 alphaonex86

Current limit is ~250 connections, but raising this limit is very high in my TODO. I'll keep you updated.

slush0 avatar Apr 23 '13 15:04 slush0

Btw this is limitation of Stratum sub-connections, you can try to run miners on getwork protocol until I fix it.

slush0 avatar Apr 23 '13 15:04 slush0

Have you more details for that's? --fix-protocol + http://X.X.X.X:8332 ? I'm very low to bandwith (less than 1% of my server bandwith)

alphaonex86 avatar Apr 23 '13 15:04 alphaonex86

Yes, it should be enough.

slush0 avatar Apr 23 '13 16:04 slush0

I have do multiple server instance (200 server) to have a quick and better fix, seam be correct?

alphaonex86 avatar Apr 23 '13 16:04 alphaonex86

Yes, you can run multiple instances of mining proxy, for example behind haproxy.

slush0 avatar Apr 23 '13 16:04 slush0

Yes but it's http proxy with load balancing, no stratum proxy.

alphaonex86 avatar Apr 23 '13 16:04 alphaonex86

haproxy is TCP proxy as well, I'm using it for balancing on the pool.

slush0 avatar Apr 23 '13 16:04 slush0

I can use haproxy as front end, and cascade of stratum proxy (255 stratum proxy connected on one stratum proxy connected on my worker?) With cgminer 2.11.3 as client, I try know if I need enable: --no-midstate --real-target It's not very clear.

alphaonex86 avatar May 02 '13 12:05 alphaonex86

no-midstate and real-target are used only for getwork. If you connects cgminer to stratum proxy with stratum protocol, you don't need to worry about this.

slush0 avatar May 02 '13 12:05 slush0

With casade for stratum proxy, it not work. When do you think it's solved? I lost 6x of my income...

alphaonex86 avatar May 03 '13 11:05 alphaonex86

https://github.com/invl/stratum-mining-proxy It's a good start I think to have 32Bits, no?

alphaonex86 avatar May 03 '13 12:05 alphaonex86

This is occuring right now for me on the WeMineLTC modified proxy. I'm only using three clients on a LAN (the proxy is providing getwork to give-me-ltc stratum), it's occuring when I connect to stratum port. Connecting to getwork port works fine. There aren't that many active ports (netstat displays 5 on :getwork, 0 on :stratum).

soundasleep avatar May 09 '13 08:05 soundasleep

Got an update on this one? I got the same issue?

markudevelop avatar May 16 '13 19:05 markudevelop

slush0: haproxy loop (connect, disconnect) each second to ping the tcp server. That's reveal a bug with your stratum proxy: more and more shares is rejected, after a 24, near 90% is rejected (mostly bellow difficulty) until I restart the server.

alphaonex86 avatar May 16 '13 21:05 alphaonex86

@alphaonex86 hey had any luck fixing this? Using the invl stratum proxy it should be around up to 65k workers for stratum but I get issues after 2k peers not sure if it's server fault or script. Did you get something working?

markudevelop avatar Jun 06 '13 21:06 markudevelop

Same, after 1k of peers I have issue and I need restart by cron each 6h... but the code seam ok. In my case other service (apache, lighttpd) have 20k peers without problem (sysctl linux tuned).

alphaonex86 avatar Jun 06 '13 23:06 alphaonex86

i have arranged it by haproxy for more then 40 instances of stratum proxy for loadbalance,it could hold more then 10000 clients on a server with 2 cores and 2GB memory.

haoyixin avatar Jun 21 '13 04:06 haoyixin

As GIL working,one instance only works on one core,so many instances will work more effectively.

haoyixin avatar Jun 21 '13 04:06 haoyixin

Any news about increase maximum connection to strtum proxy?

propostaff avatar Jul 24 '13 03:07 propostaff

I'm getting this issue on the latest proxy even though it's supposed to handle more then 250. I have less than 250 miners but it seems stale connections might not be removed maybe?

sterion75 avatar Jan 10 '14 09:01 sterion75

sterion75: Proxy prints number of currently connected peers to console. Can you please provide such log, so I can identify the problem? In my laboratory tests proxy succesfully handled hundreds of live stratum connections.

slush0 avatar Jan 21 '14 20:01 slush0

It only does that when the proxy first starts. It normally will say around 112. I don't see it mention amount of connections at any other point? Am I able to log this some how?

sterion75 avatar Jan 22 '14 11:01 sterion75

changed 1 times 2014-01-24 16:22:02,960 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:02,960 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,976 INFO stats stats.print_stats # 4 peers connected, state changed 1 times 2014-01-24 16:22:02,976 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:02,976 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,990 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:02,990 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:02,990 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,006 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,006 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,023 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,038 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,038 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,131 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,147 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,147 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,147 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,194 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,210 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,210 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,210 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,302 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,319 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,319 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,335 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,490 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,506 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,522 INFO stats stats.print_stats # 3 peers connected, state changed 1 times 2014-01-24 16:22:03,522 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,522 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,522 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,536 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,536 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2014-01-24 16:22:03,584 INFO stats stats.print_stats # 2 peers connected, state changed 1 times 2014-01-24 16:22:03,599 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,599 ERROR protocol protocol.process_failure # [Failure insta nce: Traceback: <type 'exceptions.Exception'>: Extranonce slots are full, please disconnect some miners! stratum\services.pyo:81:call twisted\internet\defer.pyo:134:maybeDeferred stratum\services.pyo:78:_run twisted\internet\defer.pyo:1187:unwindGenerator --- --- twisted\internet\defer.pyo:1045:_inlineCallbacks mining_libs\stratum_listener.pyo:138:subscribe mining_libs\stratum_listener.pyo:108:_get_unused_tail ] None 2014-01-24 16:22:03,599 INFO stats stats.print_stats # 1 peers connected, state changed 1 times

sterion75 avatar Jan 24 '14 16:01 sterion75

Hm, proxy thinks there're only two peers connected, so this really looks like a bug. I'm reading the code again and again and I don't see any edge case when this should appear...

Output of modabekov's command would be useful to see how many connections are real...

slush0 avatar Feb 10 '14 14:02 slush0

I use haproxy infront of multiple instances of the stratum proxy to support high loads

stephenc01 avatar Feb 11 '14 13:02 stephenc01

Yes, haproxy works nicely, but looks that @sterion75 hit some weird bug in proxy and I'd like to know what's wrong there.

slush0 avatar Feb 11 '14 15:02 slush0