factomd `chain-head` returns `chaininprocesslist`==true on block N+1, where N – block, in which chain included

[ilzheev]

Step 1: Chain & few entries in it created on block N.

chain-head call at 0-10 min on block N returns:

chaininprocesslist: true

Step 2: Then, chain & all entries included in block N. No new entries submitted on block N+1. chain-head call at 0-10 min on block N+1 returns:

chainhead: <hash>
chaininprocesslist: true

Step 3: chain-head call at 0-10 min on block N+2 returns:

chainhead: <hash>
chaininprocesslist: false

[ilzheev]

Here is a screenshot of chain-head factomd API call made on block N+1 (Step 2 above).

[ilzheev]

Test 2 with addition of new entry into existing chain.

Step 1 (block N): New entry created on block N. chain-head call at 0-10 min on block N returns:

        "chaininprocesslist": true,
        "chainhead": "82351b9834ad93951e01f64bf639c09ae2a609463b26cdaac2dc7bbf4e4c4a8a"

Step 2 (block N+1): Entry included in block N. No new entries submitted on block N+1. chain-head call at 0-10 min on block N+1 returns:

        "chaininprocesslist": true,
        "chainhead": "6bde150c39315f07ee09dde61a71a4a0670e592f57c40e43ff261242fd3cdae6"

Step 3 (block N+2): chain-head call at 0-10 min on block N+2 returns:

        "chaininprocesslist": false,
        "chainhead": "6bde150c39315f07ee09dde61a71a4a0670e592f57c40e43ff261242fd3cdae6"

So, the issue exists in this case too and it is not linked with creating new chain.

[WhoSoup]

Relevant code (i think) for anyone that doesn't want to go digging for it: https://github.com/FactomProject/factomd/blob/d2e5799ab97ce87030bd15ae210080a2e54ae5eb/wsapi/wsapiV2.go#L935-L942

My possible interpretation is that it checks a range of 2 blocks to cover up an error that happens if the state moves on to the next height (ie the chain is not in the processlist) but the block hasn't been saved to the database yet (ie the chain is missing from the database) in the section just below my highlight?

[PaulBernier]

Oh good thinking @WhoSoup. Then if this is to handle the edge case of minute 0 (the minute that is sealing the previous block), could the pend2 variable be replaced by a condition currentMinute == 0? Git blame indicates that @Emyrk may have some knowledge there?

[ilzheev]

I also thought maybe it was designed initially to work like this? So chain included into process list within 2 blocks... @PaulBernier @WhoSoup

Need to ask @carryforward

[carryforward]

Thank you for reporting this @ilzheev. This is related to one of the quirks of the factomd system. There is a limbo period between minute 0 and minute 1 when following along with blocks. It is kind of an uncertainty period, similar to single confirmations in bitcoin. Only after minute 1 passes does your factomd node know that the federated servers saw each other's DBsigs.

When designing the API, this level of uncertainty was not well understood and is not well articulated through the return data.
The ack call shows some of this uncertainty, with transactionack and dblockconfirmed. https://docs.factom.com/api#ack

There is an unused status where you would know if your item has been put in the process list and an EOM has been processed after your item, which would indicate that all the federated servers saw that your item had been acknowledged. That is less important that actually being saved to the database though. https://github.com/FactomProject/factomd/blob/master/common/constants/ack.go#L24

A better solution would be between minute 0 and 1, when there isn't an entry scheduled to go into the requested chain

FD-916 was made to track this ticket. I'm not sure how hard this is going to be. We will likely need a better way of of presenting uncertainty through the chain-head API in the future.