Node unable to start after available storage fills up

[cuzzo333]

Issue

Storage on my VPS was fully consumed due to a process causing 25GB+ of data to be written, possibly either due to a backup or arangodb, causing the node running on this VPS to terminate. My nodes are running on a VPS tier with a 55GB SSD and while storage utilization has increased over time, it so far has hovered around 50% utilization, so the storage filling up overnight was a bit unexpected.

I was able to free up storage by deleting a very large backup file, with approx 20GB of storage now free. When attempting to now start the node I receive the following error: warn - Failed to load contracts on all blockchain implementations. Error: Invalid JSON RPC response: ""

Up to this point I've done a bit of troubleshooting in the form of restarting my VPS, verifying configurations in the .origintrail_noderc file look fine (especially the RPC endpoint URL I have entered for Ethereum), and attempting to roll back my entire VPS to an earlier snapshot going back 4-5 days.

Steps to reproduce the problem

While I can't reproduce this issue on demand, at least one other user is also experiencing the same error after their VPS storage filled up. I had two other nodes also experience this sudden increase in storage utilization but both of those nodes are able to start after clearing up storage space.

Specifications

• Node version: v5.0.4
• Platform: N/A
• Node wallet: https://blockscout.com/xdai/mainnet/address/0x11e18f264DCB5F03e8338A9e640d0eE7CDd5408a/transactions
• ERC725 identity: 0x10c89Db8d5929B1573D68b00455A4Bd1B3186730

Contact details

• Email:

Error logs

(node:14) Warning: N-API is an experimental feature and could change at any time.
2021-06-24T19:45:34.534Z - info - npm modules dependencies check done
2021-06-24T19:45:34.535Z - info - ot-node folder structure check done
2021-06-24T19:45:34.537Z - important - Running in mainnet environment.
2021-06-24T19:45:34.561Z - info - Using existing graph database password.
2021-06-24T19:45:34.596Z - info - Arango server version 3.5.3 is up and running
2021-06-24T19:45:34.612Z - info - Storage database check done
2021-06-24T19:45:35.309Z - info - [ethr:mainnet] Selected blockchain: Ethereum
2021-06-24T19:45:35.320Z - info - [xdai:mainnet] Selected blockchain: xDai
2021-06-24T19:45:35.327Z - trace - [ethr:mainnet] Asking Hub for Holding contract address...
2021-06-24T19:45:35.359Z - trace - [xdai:mainnet] Asking Hub for Holding contract address...
**2021-06-24T19:45:55.384Z - warn - Failed to load contracts on all blockchain implementations. Error: Invalid JSON RPC response: ""**
2021-06-24 19:45:55,410 INFO exited: otnode (exit status 0; expected)
2021-06-24 19:45:56,414 WARN received SIGTERM indicating exit request
2021-06-24 19:45:56,418 INFO waiting for remote_syslog, arango, otnodelistener to die
2021-06-24 19:45:57,413 INFO stopped: arango (exit status 0)
2021-06-24 19:45:58,419 INFO stopped: remote_syslog (terminated by SIGTERM)
2021-06-24 19:45:58,420 INFO stopped: otnodelistener (terminated by SIGTERM)

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to [email protected].

[bottomshorter]

I am the other user that experienced a similar situation.

My Issue

Storage on my VPS was utilized at 96% capacity. Now when I realized this, I went to my provider to change the plan and upgrade the amount of storage available. However, the Host VM is full and I cannot upgrade the plan because of this, so I need to restore a backup and migrate it to a new machine on Vultr, or move it to DO via AWS. During the process of restoring a backup on a new node, changing the IP within the origintrail_noderc file, and relaunching the image; I was unable to get it to start and it gave me the same error.

Steps taken:

1.) Start up new instance of most recent backup from Vultr 2.) Logged into that machine 3.) Changed the noderc file and checked for syntax errors 4.) docker start otnode 5.) Results

Reproducing the problem

Just restoring a backup, and doing all the steps I did above, will give the same result. I have not tried to restore a backup of another node however.

[kotlarmilos]

It seems that you are facing an issue with RPC endpoint. Could you try to use an alternative endpoints for Ethereum and xDai and check if the issue persists?

[kotlarmilos]

Hi Tracers,

We’ve been reviewing the issues on the repo and we believe that the issue should no longer persist when pruning is enabled. However the RPC issue will be tackled by implementing multiple RPC support in the coming releases.

We will close this issue once the release with multiple RPC support is out.

[gozzynes]

Hi Guys, I have this issue currently with my node running of v5.

Failed to load contracts on all blockchain implementations. Error: Invalid JSON RPC response: ""

Is there a resolution for this issue?

Thank you for your help.

Regards.

[Valcyclovir]

There is but requires a bit of tinkering, if you want 1 on 1 help I can try lending you a hand on discord or tg node channel. @BRX86 is my handle for tg.

[NZT48]

This issue is not relevant because it was for v5 and current version of OT-node is v6.