Getting balance from node's account returns error

[Bidon15]

Celestia Node version

8e5a4ed73bd78b622b677410a69f5255db51655d

OS

Ubuntu 20.04.3 LTS / Testground

Install tools

• install TG
• clone infra repo
• pull this PR https://github.com/celestiaorg/test-infra/pull/87

Others

No response

Steps to reproduce it

Execute testground run composition -f compositions/local-docker/pfd-8.toml --wait

Expected result

Balance accounts are provided per each of DA Node types after funding is successful

Actual result

Randomly errors like:

• decoding url-encoded part #1: /-K�f�%�Jw�
• key mismatch on operation #0: expected ���Yro����dx*v��'utia but got �/���Yro����dx*v��'utia

Relevant log output

# full node decording url error
https://gist.github.com/Bidon15/92d8724d69cb9c7ced02268d2539beb0

# bridge error key mismatch
https://gist.github.com/Bidon15/dca146eb98ba10a97f5c3894c40ae528

Notes

1. Happens in state.balance that triggers celestia-core/crypto/merkle/Verify()
2. Need to be checked in k8s cluster

[renaynay]

Testing this on testground today with @Bidon15 -- app version v0.7.0-rc1 core version v1.3.1 node version latest commit 793e2b019d6d4be98bf03d91c578b3e68da13e55

first run passed (no error) second run failed for full node requesting balance:

<< fulls[000] (f2a966) >> decoding url-encoded part #1: /�tθ;k��J	%��K�ۅ2utia: invalid URL escape "%\xfa\x92"

[renaynay]

Some errors:

<< bridges[001] (de95a6) >> --- CREATED ABCI REQUEST QUERY:  {{{} [] [] <nil>} 0 [] [2 20 16 158 75 194 107 6 34 9 191 56 47 3 165 218 17 46 102 233 199 3 117 116 105 97] store/bank/key 7 true}  ---
<< bridges[001] (de95a6) >>
<< bridges[001] (de95a6) >> --- GOT ABCI RESULT:  true  ---
<< bridges[001] (de95a6) >>
<< bridges[001] (de95a6) >> --- COIN:  30000000  ---
<< bridges[001] (de95a6) >>
<< bridges[001] (de95a6) >> --- PATH TO STORE:  /bank/�K�k"	�8/��.f��utia  ---
<< bridges[001] (de95a6) >> key mismatch on operation #0: expected ��.f��utia but got �K�k"	�8/��.f��utia

<< bridges[000] (e6d2f1) >> --- CREATED ABCI REQUEST QUERY:  {{{} [] [] <nil>} 0 [] [2 20 165 249 37 16 66 143 79 14 169 25 247 43 51 111 58 65 195 117 44 57 117 116 105 97] store/bank/key 7 true}  ---
<< bridges[000] (e6d2f1) >>
<< bridges[000] (e6d2f1) >> --- GOT ABCI RESULT:  true  ---
<< bridges[000] (e6d2f1) >>
<< bridges[000] (e6d2f1) >> --- COIN:  30000000  ---
<< bridges[000] (e6d2f1) >>
<< bridges[000] (e6d2f1) >> --- PATH TO STORE:  /bank/��%B�O��+3o:A�u,9utia  ---
<< bridges[000] (e6d2f1) >> decoding url-encoded part #1: /��%B�O��+3o:A�u,9utia: invalid URL escape "%\x10B"

SUCCESS CASE for reference

<< bridges[002] (6922e2) >> --- CREATED ABCI REQUEST QUERY:  {{{} [] [] <nil>} 0 [] [2 20 251 242 148 245 178 198 76 184 98 57 247 176 130 36 53 83 27 134 95 99 117 116 105 97] store/bank/key 7 true}  ---
<< bridges[002] (6922e2) >>
<< bridges[002] (6922e2) >> --- GOT ABCI RESULT:  true  ---
<< bridges[002] (6922e2) >>
<< bridges[002] (6922e2) >> --- COIN:  30000000  ---
<< bridges[002] (6922e2) >>
<< bridges[002] (6922e2) >> --- PATH TO STORE:  /bank/������L�b9���$5SINFO	157.5966s      OTHER << bridges[002] (6922e2) >>
<< bridges[002] (6922e2) >> --- ERR FROM VERIFY VLUE ON DEFAULTPROOFRUNTIME:  <nil>  ---
<< bridges[002] (6922e2) >> bridge -> 3 has this 30000000utia balance

[renaynay]

More errors:

<< fulls[000] (80cc40) >>
<< fulls[000] (80cc40) >> --- CREATED ABCI REQUEST QUERY: data  [2 20 162 2 106 44 47 248 90 194 212 16 181 33 238 2 123 89 159 249 86 156 117 116 105 97]  --- path:  store/bank/key
<< fulls[000] (80cc40) >>
<< fulls[000] (80cc40) >> --- GOT ABCI RESULT:  true  ---
<< fulls[000] (80cc40) >>
<< fulls[000] (80cc40) >> --- COIN:  30000000  ---
<< fulls[000] (80cc40) >>
<< fulls[000] (80cc40) >> --- PATH TO STORE: formatted 0214A2026A2C2FF85AC2D410B521EE027B599FF9569C75746961, ---- unformatted: �j,/�Z���!�{Y��V�utia
<< fulls[000] (80cc40) >> --- PATH TO STORE: /bank/�j,/�Z���!�{Y��V�utia --------
<< fulls[000] (80cc40) >> --- PATH TO STORE:  /bank/�j,/�Z���!�{Y��V�utia  ---
<< fulls[000] (80cc40) >> key mismatch on operation #0: expected �Z���!�{Y��V�utia but got �j,/�Z���!�{Y��V�utia

[Bidon15]

Reproducible on upcoming upgrade of node/app

<< bridges[002] (d8ece6) >> --- CREATED ABCI REQUEST QUERY: data  [2 20 78 133 242 6 252 181 69 213 98 121 171 202 161 47 191 95 100 166 71 161 117 116 105 97]  --- path:  store/bank/key
<< bridges[002] (d8ece6) >>
<< bridges[002] (d8ece6) >> --- GOT ABCI RESULT:  true  ---
<< bridges[002] (d8ece6) >>
<< bridges[002] (d8ece6) >> --- COIN:  30000000  ---
<< bridges[002] (d8ece6) >>
<< bridges[002] (d8ece6) >> --- PATH TO STORE: formatted 02144E85F206FCB545D56279ABCAA12FBF5F64A647A175746961, ---- unformatted: N����E�by�ʡ/�_d�G�utia
<< bridges[002] (d8ece6) >> --- PATH TO STORE: /bank/N����E�by�ʡ/�_d�G�utia --------
<< bridges[002] (d8ece6) >> --- PATH TO STORE:  /bank/N����E�by�ʡ/�_d�G�utia  ---
<< bridges[002] (d8ece6) >> key mismatch on operation #0: expected �_d�G�utia but got N����E�by�ʡ/�_d�G�utia

[renaynay]

Have we seen this recently @Bidon15 ?

[Bidon15]

Have we seen this recently @Bidon15 ?

Only in testground tests so far Not in manual mode so far. Will keep posted if I can catch it

[Bidon15]

Reproducible in v0.10.0 and latest node version 😖

[Bidon15]

"key mismatch on operation #0: expected sy\u001c\u0007\u0006\ufffda\ufffd\u0002\ufffd\u0010\ufffd\ufffd\ufffd\n&\u0017\ufffdutia but got \u0002\u0014{/sy\u001c\u0007\u0006\ufffda\ufffd\u0002\ufffd\u0010\ufffd\ufffd\ufffd\n&\u0017\ufffdutia"

Checking why on latest node version and app v0.11.0 we have this error

[Bidon15]

Closing as fixed according #1483