Light client can't get state for older block

[s1na]

I have a light server and a light client running at the same time which are peered up to one-another (and nothing else). The light client cannot get the state from an older block even tho the server has that state available. See console logs:

Server:

> eth.blockNumber
15045810
> debug.getAccessibleState(eth.blockNumber-2, eth.blockNumber)
15045809
> debug.getAccessibleState(eth.blockNumber-1280, eth.blockNumber)
15045683
> debug.getAccessibleState(eth.blockNumber-12800, eth.blockNumber)
15035214
> eth.getTransactionCount('0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8', web3.toHex(15035214))
43582525

Client:

> eth.getTransactionCount('0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8', web3.toHex(15045809))
43615173
> eth.getTransactionCount('0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8', web3.toHex(15045683))
43614896
> eth.getTransactionCount('0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8', web3.toHex(15035214))
Error: getDeleteStateObject (ea674fdde714fd979de3edf0f56aa9716b898ec8) error: no suitable peers available
        at web3.js:6365:37(47)
        at send (web3.js:5099:62(35))
        at <eval>:1:24(9)

Something I noticed is the older state goes through the HelperTrie:

DEBUG[07-04|14:14:23.612] Requesting CHT                           id=17b99695e530823c conn=dyndial cht=458 block=14,996,201
DEBUG[07-04|14:14:23.612] Fetching batch of HelperTrie proofs      id=17b99695e530823c conn=dyndial count=1
TRACE[07-04|14:14:23.613] Light Ethereum message arrived           id=17b99695e530823c conn=dyndial code=18 bytes=3495
TRACE[07-04|14:14:23.613] Received helper trie proof response      id=17b99695e530823c conn=dyndial
DEBUG[07-04|14:14:23.613] Validating CHT                           cht=458 block=14,996,201
WARN [07-04|14:14:26.614] Served eth_getTransactionCount           reqid=66 duration=3.002109651s err="getDeleteStateObject (ea674fdde714fd979de3edf0f56aa9716b898ec8) erro
r: no suitable peers available"

Whereas the recent state requests:

DEBUG[07-04|16:00:51.439] Requesting trie proof                    id=17b99695e530823c conn=dyndial root=d41b85..6a060f key="[89 231 68 154 172 237 104 59 60 168 130 105 1
6 24 46 102 68 79 22 218 87 93 151 81 178 138 89 244 78 112 208 177]"
DEBUG[07-04|16:00:51.439] Fetching batch of proofs                 id=17b99695e530823c conn=dyndial count=1
TRACE[07-04|16:00:51.441] Light Ethereum message arrived           id=17b99695e530823c conn=dyndial code=16 bytes=3637
TRACE[07-04|16:00:51.441] Received les/2 proofs response           id=17b99695e530823c conn=dyndial
DEBUG[07-04|16:00:51.441] Validating trie proof                    root=d41b85..6a060f key="[89 231 68 154 172 237 104 59 60 168 130 105 16 24 46 102 68 79 22 218 87 93 15
1 81 178 138 89 244 78 112 208 177]"

[rjl493456442]

In this request

> eth.getTransactionCount('0xEA674fdDe714fd979de3EdF0F56AA9716B898ec8', web3.toHex(15035214))
43582525

Block 15035214 should be requested(because it's pruned). However from the log, it said

DEBUG[07-04|14:14:23.613] Validating CHT                           cht=458 block=14,996,201

I guess logs are not matched?

[s1na]

@rjl493456442 yeah I tried again and I'm not getting those helper trie logs again, it was probably a mistake on my part. I just get

WARN [07-06|17:45:36.915] Served eth_getTransactionCount           reqid=80 duration=3.00169405s err="getDeleteStateObject (ea674fdde714fd979de3edf0f56aa9716b898ec8) error: no suitable peers available"

even with --vmodule p2p=5,les=5,light=5.

[rjl493456442]

// Refuse to search stale state data in the database since looking for
// a non-exist key is kind of expensive.
local := bc.CurrentHeader().Number.Uint64()
if !backend.ArchiveMode() && header.Number.Uint64()+core.TriesInMemory <= local {
	p.Log().Debug("Reject stale trie request", "number", header.Number.Uint64(), "head", local)
	p.bumpInvalid()
	continue
}
root = header.Root

So In LES Server, it will only serve the latest 128 states, even it has the historic state available locally. This explains why you get this error.

[fjl]

I think we could create an option that disables this check in LES server.

[MariusVanDerWijden]

We have removed the light client recently, so I'm going to close this issue