lbry-sdk suite: test_es_sync_utility: AssertionError: 217 != 218

Open moodyjon opened this issue 3 years ago • 1 comments

It seems this is a vulnerability in the es_writer.start() where it can fail if interleaved with a task adding new block(s) (e.g. create_task(self.generate(N))). Hence filing against lbryio/hub.

Test Code:

        # stop the es writer and advance the chain by 1, adding a new claim. upon resuming the es writer, it should
        # add the new claim
        await es_writer.stop()
        await self.stream_create(f"stream11", bid='0.001', confirm=False)
        generate_block_task = asyncio.create_task(self.generate(1))
        await es_writer.start()
        await generate_block_task
        self.assertEqual(11, len(await self.claim_search(order_by=['height'])))

https://github.com/lbryio/lbry-sdk/blob/8becf1f69f38019c8c1d0ac6fbba80897f94c8ed/tests/integration/blockchain/test_wallet_server_sessions.py#L126

Failure:

======================================================================
[653](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:654)
FAIL: test_es_sync_utility (integration.blockchain.test_wallet_server_sessions.TestESSync)
[654](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:655)
----------------------------------------------------------------------
[655](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:656)
Traceback (most recent call last):
[656](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:657)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/testcase.py", line 145, in run
[657](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:658)
    self.loop.run_until_complete(maybe_coroutine)
[658](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:659)
  File "/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
[659](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:660)
    return future.result()
[660](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:661)
  File "/home/runner/work/lbry-sdk/lbry-sdk/tests/integration/blockchain/test_wallet_server_sessions.py", line 131, in test_es_sync_utility
[661](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:662)
    await es_writer.start()
[662](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:663)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 389, in start
[663](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:664)
    return await super().start()
[664](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:665)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/service.py", line 81, in start
[665](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:666)
    await start_task
[666](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:667)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 315, in catch_up
[667](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:668)
    self.advance(height)
[668](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:669)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/elastic_sync/service.py", line 230, in advance
[669](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:670)
    super().advance(height)
[670](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:671)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/blockchain/lib/python3.9/site-packages/hub/service.py", line 159, in advance
[671](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:672)
    assert len(self.db.tx_counts) == height, f"{len(self.db.tx_counts)} != {height}"
[672](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:673)
AssertionError: 217 != 218
[673](https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:674)

Examples: https://github.com/lbryio/lbry-sdk/runs/6944766671?check_suite_focus=true#step:11:686 https://github.com/lbryio/lbry-sdk/runs/6846528407?check_suite_focus=true#step:11:499 https://github.com/lbryio/lbry-sdk/runs/6846200096?check_suite_focus=true#step:11:469

Jun 19 '22 23:06 moodyjon

Thanks for pointing this out, this appears to be a race condition in the tests (it is not a vulnerability, just a bug) - these commits are working towards fixing it:

https://github.com/lbryio/hub/pull/57/commits/8fe3e0915df62a0b7dff67d7d3368938b4e0a69e https://github.com/lbryio/lbry-sdk/commit/cdb9480e89725ef9b171b132bcaf7726fdf9f93c

Jun 29 '22 16:06 jackrobison