steem-python icon indicating copy to clipboard operation
steem-python copied to clipboard

stream_from occasionally returns None

Open roadscape opened this issue 8 years ago • 1 comments

Example:

from steem.blockchain import Blockchain
from steem.steemd import Steemd

steemd = Steemd()
b = Blockchain(mode='head')
h = b.stream_from(
    start_block=14500000,
    full_blocks=True,
)
for block in h:
    if not block or 'block_id' not in block:
        raise Exception("invalid block: {}".format(block))

    num = int(block['block_id'][:8], base=16)
    print("Got block {} at {} with {} txs -- ".format(num,
        block['timestamp'], len(block['transactions'])))

Will sometimes fail like:

Got block 14577978 at 2017-08-14T21:41:36 with 19 txs
Got block 14577979 at 2017-08-14T21:41:39 with 17 txs
Got block 14577980 at 2017-08-14T21:41:42 with 15 txs
Traceback (most recent call last):
  File "hive/indexer/core.py", line 412, in <module>
    run()
  File "hive/indexer/core.py", line 400, in run
    listen_steemd()
  File "hive/indexer/core.py", line 354, in listen_steemd
    raise Exception("stream returned invalid block: {}".format(block))
Exception: stream returned invalid block: None

stream_from should never return None. If there is some issue it should throw an exception.

Output from the node indicates this case may have been due to double-production:

2502307ms th_a       application.cpp:507           handle_block         ] Got 15 transactions on block 14577980 by xeldal -- latency: 307 ms
2503812ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc": "2.0", "id": 0, "params": ["database_api", "get_dynamic_global_properties", []]
2503813ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc": "2.0", "id": 0, "params": ["database_api", "get_block", [14577980]], "method": "call"}
2505942ms th_a       application.cpp:507           handle_block         ] Got 20 transactions on block 14577981 by fubar-bdhr -- latency: 942 ms
2505995ms th_a       database.cpp:538              _maybe_warn_multiple ] Encountered block num collision at block 14577981 due to a fork, witnesses are: [["fubar-bdhr","2017-08-14T21:41:45"],["fubar-bdhr","2017-08-14T21:41:45"]]
2505996ms th_a       application.cpp:507           handle_block         ] Got 20 transactions on block 14577981 by fubar-bdhr -- latency: 996 ms
2506977ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc": "2.0", "id": 0, "params": ["database_api", "get_dynamic_global_properties", []], "method": "call"}
2506978ms th_a       websocket_api.cpp:88          on_message           ] message: {"jsonrpc": "2.0", "id": 0, "params": ["database_api", "get_block", [14577981]], "method": "call"}

roadscape avatar Aug 14 '17 22:08 roadscape

Fixed in https://github.com/Netherdrake/steem-python/commit/3f0a8b5e406a892609990ab5932ccce5a893eb1b

Netherdrake avatar Aug 15 '17 00:08 Netherdrake