holochain
`hdk::link_entries` fails with faulty "Base for link not found" w/ >2 `hc run` agents
HoloFuel fails w/ a HolochainError::ErrorGeneric, when more than 3 Agents are run under hc run ..., with n3h:
"HoloFuel Link \"promise_preauth\" from Agent HcSCJnb8mm9YOxu8hsSyj5aKzB9Dantg5Fdz3AxAP7J3f9movrkYmjIAtxwx6si to commit QmQRCaEW5EGJrmNf4Vpne4bczVFsGx1hB33sBgrNf9Vtmz failed: {\"kind\":{\"ErrorGeneric\":\"Base for link not found\"},\"file\":\"core/src/nucleus/ribosome/runtime.rs\",\"line\":\"211\"}"
It appears that all of the Agent IDs are not being percolated to all DHT nodes, or they are otherwise somehow not being found by the hdk::link_entries call. This defect does not seem to occur under nodejs_conductor (a unit test of this scenario with 3 agents runs reliably), but it always occurs when 3 agents are run under hc run ....
I've attached 3 hc run --logging ... outputs. The dna.log.bootstrap and dna.log.agent1 nodes commit and link some Requests for funds from agent2. The dna.log.agent2' attempts to commit the corresponding Promises to each of the bootstrap and agent1 Agents; it succeeds with one (the bootstrap node in this case, but it appears random) -- but it fails to successfully call hdk::link_entries with the base Agent ID of agent1.
The bootstrap Agent ID: HcSCJvyu5aUXiyvubmkfapFJmHq6fxfojfFWqc9w4Iyxdq4sGQej5HY47hNte6i The agent1 Agent ID: HcSCJnb8mm9YOxu8hsSyj5aKzB9Dantg5Fdz3AxAP7J3f9movrkYmjIAtxwx6si The agent2 Agent ID: HcScjC6dcu593Y6YwjbMHnp4ASRx7p8mijbnuVOJDjntc5inS54mihVqSmavtpr
The Promise being committed is correct ("base" is Agent1's ID: HcSCJn...wx6si). This exists in the DHT, but won't have previously been the "base" of an hdk::link_entries call.
Hi, @lucksus -- you were asking about confirming that this issue actually exists, and whether or not nodejs_conductor exhibits the issue.
The answer is yes, it seems to be a legitimate issue w/ hc run ... and n3h with 3 agents, and no, nodejs_conductor doesn't seem to suffer from this issue with 3 agents.
Appears to be some kind issue where the Agent ID of one of the first 2 agents doesn't propogate to the 3rd agent, so when it tries to do an hdk::link_entries one of the first two agent's Agent ID isn't a valid "base" for the link...
Here's the actual Promise that was committed, showing its from/to Agent IDs (the to: agent ID was the base of the link):
{
"request":"Qmcn31LsyE83eCQ2bfrbPyu79WhoZkL1vqjk3TQ9YYSTbX",
"from":"HcScjC6dcu593Y6YwjbMHnp4ASRx7p8mijbnuVOJDjntc5inS54mihVqSmavtpr",
"to":"HcSCJnb8mm9YOxu8hsSyj5aKzB9Dantg5Fdz3AxAP7J3f9movrkYmjIAtxwx6si",
"amount":"1.1",
"fee":"0.011",
"deadline":"2019-04-17T19:45:20.033+00:00",
"notes":"from agent 1"
}
On further diagnosis in holochain-rust develop branch, the problem persists, and seems to indicate that the 3rd agent added can only successfully communicate with one of the original agents. By inspecting the logs, you can observe errors like this:
2019-04-26 12:42:31:test-instance: err/workflow/hold_entry: QmYYnoyJouU7Hs98FsVY1Cq5BVi93Ps43ghJrB2TLTWb5E is NOT valid! UnresolvedDependencies([HashString("HcSCJvyu5aUXiyvubmkfapFJmHq6fxfojfFWqc9w4Iyxdq4sGQej5HY47hNte6i"), HashString("QmUL1ddqkgm4md2aCpBTUQabx3doiPD4TN9KUn7u8DKzge")])
2019-04-26 12:42:31:test-instance: err/workflow/hold_entry: Qmbs5ukSkvdDFj8kBfjimtkHQnEUmdu6HTWBqY3TFJCRqQ is NOT valid! UnresolvedDependencies([HashString("HcSCJvyu5aUXiyvubmkfapFJmHq6fxfojfFWqc9w4Iyxdq4sGQej5HY47hNte6i"), HashString("QmeiD8AUD3tpH92bgdGbyzMpKonPTxr4UxqmRZn3PUCCN3")])
By avoiding the Agent ID that is in error, communications with the other agent ID appear to work normally, for the most part -- I may have observed a situation where setting links with the operational counterparty agent ID as the "base" worked, and then later ceased to work, but I cannot be certain I executed the test as I recall.
@pjkundert are you still able to repro this issue, and if so, may you add specific steps how you would reproduce? (For instance, I've never used HoloFuel before, read its docs, etc.)
