ot-node icon indicating copy to clipboard operation
ot-node copied to clipboard

Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id

Open Valcyclovir opened this issue 3 years ago • 2 comments

Issue description

My node starts but goes into reboot loop continuously

Expected behavior

Node starts and keeps running

Actual behavior

Reboot loop after successfully starting the node and every dependencies

Steps to reproduce the problem

  1. Install a fresh node
  2. check logs after every dependencies load

Specifications

  • Node version: v6.0.0-beta.2.2.2
  • Platform: Ubuntu 20.04
  • Node wallet: 0xf19da42c2fa6070d197c53a95cd5d8999696cabf
  • Node libp2p identity: Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M

The same thing happened to my arch linux node running Kernel 5.19.6

Contact details

Error logs

[2022-09-03 05:29:09] DEBUG: Marking operation id 0ecd3475-7faa-46e2-8628-e69a2aa7f346 as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: 0ecd3475-7faa-46e2-8628-e69a2aa7f346 cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/0ecd3475-7faa-46e2-8628-e69a2aa7f346 [2022-09-03 05:29:09] DEBUG: Removing operation id: 0ecd3475-7faa-46e2-8628-e69a2aa7f346 cached data from memory [2022-09-03 05:29:09] WARN: Error in command: publishInitCommand, error: id.equals is not a function [2022-09-03 05:29:09] WARN: Error in command: publishInitCommand, error: id.equals is not a function [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to QmUUf9WhoTK7sbbfZsQVFjLEnW3GwF5Asiycooas2udvyk: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: cac2f024-7c6f-4d95-be43-8f0fd6798bca [2022-09-03 05:29:09] DEBUG: Marking operation id cac2f024-7c6f-4d95-be43-8f0fd6798bca as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: cac2f024-7c6f-4d95-be43-8f0fd6798bca cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/cac2f024-7c6f-4d95-be43-8f0fd6798bca [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to QmadRL22p2ipwS44DisDtwWiGTpx6NtTFkZFuUh97HqpqD: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: f828b454-594a-4275-9066-fcce7fd6f7ad [2022-09-03 05:29:09] DEBUG: Marking operation id f828b454-594a-4275-9066-fcce7fd6f7ad as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: f828b454-594a-4275-9066-fcce7fd6f7ad cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/f828b454-594a-4275-9066-fcce7fd6f7ad [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to Qma5qcgyM826rHgczt5SKfseyrHDDr3iCcxKWSYwyPfuh2: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: 216f58f8-682c-44cf-adfe-818cb8be0c71 [2022-09-03 05:29:09] DEBUG: Marking operation id 216f58f8-682c-44cf-adfe-818cb8be0c71 as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: 216f58f8-682c-44cf-adfe-818cb8be0c71 cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/216f58f8-682c-44cf-adfe-818cb8be0c71 [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to QmXFXCffcrTZR2mgKYH4ArvSmDkb7YzeHb28MfokRjfEqV: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: fdab6a14-72be-4274-9cca-878751e0de5c [2022-09-03 05:29:09] DEBUG: Marking operation id fdab6a14-72be-4274-9cca-878751e0de5c as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: fdab6a14-72be-4274-9cca-878751e0de5c cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/fdab6a14-72be-4274-9cca-878751e0de5c [2022-09-03 05:29:09] DEBUG: Removing operation id: cac2f024-7c6f-4d95-be43-8f0fd6798bca cached data from memory [2022-09-03 05:29:09] DEBUG: Removing operation id: f828b454-594a-4275-9066-fcce7fd6f7ad cached data from memory [2022-09-03 05:29:09] DEBUG: Removing operation id: 216f58f8-682c-44cf-adfe-818cb8be0c71 cached data from memory [2022-09-03 05:29:09] DEBUG: Removing operation id: fdab6a14-72be-4274-9cca-878751e0de5c cached data from memory [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to QmPzKAXVZoNkhHRQp6ai7WXnN8f5w4BfgbmoUnXqDihF2z: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: cb82237f-4325-496e-bd82-f9fae2cdf12d [2022-09-03 05:29:09] DEBUG: Marking operation id cb82237f-4325-496e-bd82-f9fae2cdf12d as failed [2022-09-03 05:29:09] DEBUG: Removing operation id: cb82237f-4325-496e-bd82-f9fae2cdf12d cached data [2022-09-03 05:29:09] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/cb82237f-4325-496e-bd82-f9fae2cdf12d Something went really wrong! OT-node shutting down... [Error: ENOENT: no such file or directory, unlink '/root/ot-node/data/operation_id_cache/a425b55c-0e08-4509-8c65-48008e10f143'] { errno: -2, code: 'ENOENT', syscall: 'unlink', path: '/root/ot-node/data/operation_id_cache/a425b55c-0e08-4509-8c65-48008e10f143' } [2022-09-03 05:29:09] DEBUG: Sending response from Qmdki3QeKaQGGowTYz5v59vhz5b6S9R9NouANjuiZc4y7M to QmZYXTRArxYTo9skNtV7zFzQPkc4xrS1ZZdX8s1LzLHYsp: event=/store/1.0.0, messageType=ACK; [2022-09-03 05:29:09] ERROR: Command error (PublishRemoteError): Retry count for command: handleStoreInitCommand reached! Unable to validate data for operation id: a425b55c-0e08-4509-8c65-48008e10f143 [2022-09-03 05:29:09] DEBUG: Marking operation id a425b55c-0e08-4509-8c65-48008e10f143 as failed otnode.service: Main process exited, code=exited, status=1/FAILURE otnode.service: Failed with result 'exit-code'. otnode.service: Scheduled restart job, restart counter is at 5. Stopped OriginTrail V6 Stage 1 Beta Node. Started OriginTrail V6 Stage 1 Beta Node.

Full logs: log.txt

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to [email protected].

Valcyclovir avatar Sep 03 '22 05:09 Valcyclovir

I'm getting the same thing I'm not trying to publish anything, I've rebooted and restarted the node, with no changes It's now been doing this for hours

DalSlacker avatar Sep 07 '22 01:09 DalSlacker

Also getting this error

[2022-09-09 11:43:33] DEBUG: Removing file on path: /root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b Something went really wrong! OT-node shutting down... [Error: ENOENT: no such file or directory, unlink '/root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b'] { errno: -2, code: 'ENOENT', syscall: 'unlink', path: '/root/ot-node/data/operation_id_cache/8f86246b-59b4-468b-be3a-a4c0b6d6eb8b' } otnode.service: Main process exited, code=exited, status=1/FAILURE otnode.service: Failed with result 'exit-code'. otnode.service: Scheduled restart job, restart counter is at 8203. Stopped OriginTrail V6 Stage 1 Beta Node. Started OriginTrail V6 Stage 1 Beta Node.

It ends up in an infinite loop of this

Baddooo14 avatar Sep 10 '22 12:09 Baddooo14

This issue is being closed as inactive due to the date of the last activity on it. If you believe this is still a problem, please create a new issue and confirm that it is reproducible in the current ot-node release version. We are working towards closing open issues that meet specific criteria and ask you to create a new one for those that that are truly bugs in current release. We'll be monitoring those issues so that they are properly managed.

Thank you, OriginTrail Team

NZT48 avatar Dec 26 '22 16:12 NZT48