Can't sync and start chaos (staging-fast-active-bellatrix)

[tmeinlschmidt]

Describe the bug After various attemps and config chages it started to sync, but ended up with errors

2024-01-05 23:30:42.932137   ... download progress ... 52 of 53
2024-01-05 23:30:42.932292   Snapshot download success for block 2613049

2024-01-05 23:30:42.932658   FATAL: Exception while processing downloaded snapshot:  --- FATAL: Exception while importing downloaded snapshot:  --- Snapshot for block 2613049 is already present ---
2024-01-05 23:30:42.935568   CRITICAL FATAL: tried to download snapshot from everywhere! ---
2024-01-05 23:30:42.935606
  /skaled/skaled : ()+0x717ccd
  /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xe7
  /skaled/skaled : _start()+0x2a
...
2024-01-05 23:38:50.149991   CRITICAL Error creating blockchain --- /home/ubuntu/actions-runner/_work/skaled/skaled/libdevcore/LevelDB.cpp(60): Throw in function void dev::db::{anonymous}::checkStatus(const leveldb::Status&, const boost::filesystem::path&)
Dynamic exception type: boost::exception_detail::clone_impl<dev::db::DatabaseError>
[dev::db::tag_dbStatusCode*] = [ type: dev::db::DatabaseStatus, size: 4, dump: 02 00 00 00 ]
[dev::db::tag_dbStatusString*] = Corruption: corrupted compressed block contents
 ---
2024-01-05 23:38:50.150246
  /skaled/skaled : ()+0x717ccd
  /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xe7
  /skaled/skaled : _start()+0x2a

To Reproduce env file:

ENFORCE_BTRFS="True"
CONTAINER_CONFIGS_STREAM="2.1.15-sync-2.0-pause-hotfix"
ENDPOINT="https://rpc.ankr.com/eth_goerli"
MANAGER_CONTRACTS_ABI_URL="https://raw.githubusercontent.com/skalenetwork/skale-network/master/releases/staging-v3/skale-manager/1.9.2/manager.json"
IMA_CONTRACTS_ABI_URL="https://raw.githubusercontent.com/skalenetwork/skale-network/master/releases/staging-v3/IMA/1.3.4-rc.0/mainnet/ima.json"
DISK_MOUNTPOINT=/dev/sdb
DOCKER_LVMPY_STREAM="1.0.2-stable.0"
ENV_TYPE="testnet"
SCHAIN_NAME="staging-fast-active-bellatrix"

skale version: SKALE Node CLI version: 2.3.0-sync-node.0 ubuntu: Ubuntu 20.04.6 LTS

if I try the same with skale 2.2.0, I'm getting this error

Requirement already satisfied: six>=1.5 in ./venv/lib/python3.8/site-packages (from python-dateutil->python-crontab==2.6.0->-r requirements.txt (line 6)) (1.16.0)
Lvmpy cleanup finished
Enabling service
Service is up
Checking driver health
Traceback (most recent call last):
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
TypeError: request() got an unexpected keyword argument 'chunked'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "health.py", line 198, in <module>
    main()
  File "health.py", line 183, in main
    pc = PreinstallCheck(
  File "health.py", line 70, in __init__
    self.client = docker.from_env()
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/opt/docker-lvmpy/venv/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'

Expected behavior node synced up and running

[badrogger]

Hello @tmeinlschmidt. Thank you for rasing the issue. Could you please try 2.3.1 version of node-cli and CONTAINER_CONFIG_STREAM=2.2.1-sync

[tmeinlschmidt]

hi, trying, 2.3.1 has no sync-node command

root:~# skale sync-node init env
Usage: skale [OPTIONS] COMMAND [ARGS]...
Try 'skale --help' for help.

Error: No such command 'sync-node'.

when trying skale node init env, getting

root:~# skale node init env
Your env file(env) have some absent params: IMA_ENDPOINT, FILEBEAT_HOST, SGX_SERVER_URL.
You should specify them to make sure that all services are working