ion icon indicating copy to clipboard operation
ion copied to clipboard

Conflicts on transaction number causing unresolved DID's?

Open tonyimpervious opened this issue 2 years ago • 11 comments

Can't tell why I can't reliably create testnet DID's from my own node.

I'm often seeing (and not just mine) multiple anchor operations reference the same transaction number but only the first one that shows up in the list will end up being what's resolved.

Example:

Sidetree transaction found; adding {"transactionNumber":9047095205953548,"transactionTime":2106441,"transactionTimeHash":"0000000000000015f744b5ce9b6188b05897da187e45271fee16b8226ef2046a","anchorStrin
g":"2.QmW184wFhuJk1vfywoZDbKET7JFABaACLRA4oizMnxdtLA","transactionFeePaid":1362,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}                                                                    
Sending jRPC request: id: 4lo5cjnersj, method: getrawtransaction                                                                                                                                        
Sending jRPC request: id: 3uiu4kc6dvg, method: getrawtransaction                                                                                                                                        
Sending jRPC request: id: 3oqk3ie1q5q, method: getrawtransaction                                                                                                                                        
Sidetree transaction found; adding {"transactionNumber":9047095205953548,"transactionTime":2106441,"transactionTimeHash":"0000000000000015f744b5ce9b6188b05897da187e45271fee16b8226ef2046a","anchorStrin
g":"2.Qmc5wTAzNMf12mYBfuMuo9w5bKWcvTXkVQNYPKqC7veMGf","transactionFeePaid":1362,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}      

Both have 9047095205953548 from block 2106441 yet when I look into the mongo DB, I just see:

ion-testnet-bitcoin/transactions

{ 
    "_id" : ObjectId("61ae8db7c3d8ef289197b041"), 
    "anchorString" : "6.Qmawo2ZhRYcZNJYWD1RLg5DzzZVrMvhbVoi5SRP5K5tfN7", 
    "transactionNumber" : NumberLong(9047099500920836), 
    "transactionTime" : NumberInt(2106442), 
    "transactionTimeHash" : "0000000000000013fe2525b778c271ebd03fa0b13bb35f0b2eba3abf0a5760b1", 
    "transactionFeePaid" : NumberInt(1361), 
    "normalizedTransactionFee" : null, 
    "writer" : "395eb64155351b47dad51bc8da3675b1231b18d4"
}
{ 
    "_id" : ObjectId("61ae8c12c3d8ef289197b03f"), 
    "anchorString" : "2.QmW184wFhuJk1vfywoZDbKET7JFABaACLRA4oizMnxdtLA", 
    "transactionNumber" : NumberLong(9047095205953548), 
    "transactionTime" : NumberInt(2106441), 
    "transactionTimeHash" : "0000000000000015f744b5ce9b6188b05897da187e45271fee16b8226ef2046a", 
    "transactionFeePaid" : NumberInt(1362), 
    "normalizedTransactionFee" : null, 
    "writer" : "7628046f6ec479c031d03a47b6a003c4dedb1cad"
}
{ 
    "_id" : ObjectId("61ae8816c3d8ef289197b03e"), 
    "anchorString" : "1.QmPkgXKo38RKgfB2koNQ3rucL7J2pELuFzt9em6aCWQJ92", 
    "transactionNumber" : NumberLong(9047086616018952), 
    "transactionTime" : NumberInt(2106439), 
    "transactionTimeHash" : "0000000000000017b75575af81933bc7b4bfa7443047b7d60d75f6c85ec7370f", 
    "transactionFeePaid" : NumberInt(1362), 
    "normalizedTransactionFee" : null, 
    "writer" : "7628046f6ec479c031d03a47b6a003c4dedb1cad"
}

ion-testnet-core/transactions

{ 
    "_id" : ObjectId("61ae8e00b58eb9293eaf9a44"), 
    "anchorString" : "6.Qmawo2ZhRYcZNJYWD1RLg5DzzZVrMvhbVoi5SRP5K5tfN7", 
    "transactionNumber" : NumberLong(9047099500920836), 
    "transactionTime" : NumberInt(2106442), 
    "transactionTimeHash" : "0000000000000013fe2525b778c271ebd03fa0b13bb35f0b2eba3abf0a5760b1", 
    "transactionFeePaid" : NumberInt(1361), 
    "normalizedTransactionFee" : NumberInt(1348), 
    "writer" : "395eb64155351b47dad51bc8da3675b1231b18d4"
}
{ 
    "_id" : ObjectId("61ae8c55b58eb9293eaf9a43"), 
    "anchorString" : "2.QmW184wFhuJk1vfywoZDbKET7JFABaACLRA4oizMnxdtLA", 
    "transactionNumber" : NumberLong(9047095205953548), 
    "transactionTime" : NumberInt(2106441), 
    "transactionTimeHash" : "0000000000000015f744b5ce9b6188b05897da187e45271fee16b8226ef2046a", 
    "transactionFeePaid" : NumberInt(1362), 
    "normalizedTransactionFee" : NumberInt(1348), 
    "writer" : "7628046f6ec479c031d03a47b6a003c4dedb1cad"
}
{ 
    "_id" : ObjectId("61ae888eb58eb9293eaf9a41"), 
    "anchorString" : "1.QmPkgXKo38RKgfB2koNQ3rucL7J2pELuFzt9em6aCWQJ92", 
    "transactionNumber" : NumberLong(9047086616018952), 
    "transactionTime" : NumberInt(2106439), 
    "transactionTimeHash" : "0000000000000017b75575af81933bc7b4bfa7443047b7d60d75f6c85ec7370f", 
    "transactionFeePaid" : NumberInt(1362), 
    "normalizedTransactionFee" : NumberInt(1348), 
    "writer" : "7628046f6ec479c031d03a47b6a003c4dedb1cad"
}

ion-testnet-core/operations

{ 
    "_id" : ObjectId("61ae8dcccfd5dc4f511063af"), 
    "type" : "create", 
    "didSuffix" : "EiAhioA-4dtWS250BqxqdXz2H575TBpHMTZHf78C7-JWmw", 
    "operationBufferBsonBinary" : BinData(0, "eyJ0eXBlIjoiY3JlYXRlIiwic3VmZml4RGF0YSI6eyJkZWx0YUhhc2giOiJFaURYTExjMmpXX3Z1cW9fOENhNktxUFRDdGhoeWQxTXhrN2kxNVRxNnFOQzd3IiwicmVjb3ZlcnlDb21taXRtZW50IjoiRWlDcXJiVHJ6dFQ5VVRhREZRVUxCcXNVRG0zQVplWkNadHpZT2V1ZDBpWmZyZyJ9LCJkZWx0YSI6eyJwYXRjaGVzIjpbeyJhY3Rpb24iOiJyZXBsYWNlIiwiZG9jdW1lbnQiOnsicHVibGljS2V5cyI6W3siaWQiOiJrZXktMSIsInB1YmxpY0tleUp3ayI6eyJrdHkiOiJPS1AiLCJjcnYiOiJFZDI1NTE5IiwieCI6InRwUFNRN3dYa0RVb1BCNzR2ZVM0WnF1c194VGVicEZCNmh5OEFSb0NySGMifSwicHVycG9zZXMiOlsiYXV0aGVudGljYXRpb24iXSwidHlwZSI6Ikpzb25XZWJLZXkyMDIwIn1dLCJzZXJ2aWNlcyI6W3siaWQiOiJzY2hlbWEtMSIsInR5cGUiOiJzY2hlbWEiLCJzZXJ2aWNlRW5kcG9pbnQiOiJkaWQ6d29yazpWdzFicXlwcmg3NmVWUVZraDNuNGV2O2lkPTE4NGUzMDE1LTMwNTUtNGUwNy04YTg2LTViYjQyMGVlODgzNTt2ZXJzaW9uPTEuMCJ9XX19XSwidXBkYXRlQ29tbWl0bWVudCI6IkVpQXNlaDdIR3FUVFVPdEtYVlpNYmpoMWxoYU5kY3dUeXBTbmdCY2M1REUtbUEifX0="), 
    "opIndex" : NumberInt(5), 
    "txnNumber" : NumberLong(9047099500920836), 
    "txnTime" : NumberInt(2106442)
}
{ 
    "_id" : ObjectId("61ae8c5bcfd5dc4f51105fed"), 
    "type" : "create", 
    "didSuffix" : "EiA1keteQaNuo2B_9-HO6BPN_8KVfa56xvRnCih4QW11Ww", 
    "operationBufferBsonBinary" : BinData(0, "eyJ0eXBlIjoiY3JlYXRlIiwic3VmZml4RGF0YSI6eyJkZWx0YUhhc2giOiJFaUNnaXNUVV9acE9mcTNFZERSOHFfaF9EdjduNElKMmxQN09xZEtOQjVhbUhBIiwicmVjb3ZlcnlDb21taXRtZW50IjoiRWlCZnRqTkZ3NTYzT0t3dFdlUmR0dGllSTU0cEJycnM1QmUwMU5hWmNFVm53dyJ9LCJkZWx0YSI6eyJ1cGRhdGVDb21taXRtZW50IjoiRWlEcnlqTzVfMzJ3YVFOUVZoMmVRV1RGZXF4b0huaEwyZ29NamZxdnNJRjlpZyIsInBhdGNoZXMiOlt7ImFjdGlvbiI6InJlcGxhY2UiLCJkb2N1bWVudCI6eyJwdWJsaWNLZXlzIjpbeyJpZCI6InNpZ19hNWExMWYzZSIsInB1YmxpY0tleUp3ayI6eyJrdHkiOiJFQyIsImNydiI6InNlY3AyNTZrMSIsIngiOiI2UFJJZlBzQmg3SVp5ZnV0cm5rZmZwY0tPQWZpQTZTYnU2RTB3YU8tQmprIiwieSI6IlZnRS12MmZzQUlYRDV6Y2d1MjFWTGFRVXJiaXMyYTRtdGdTeHRQRmNrNUkifSwidHlwZSI6IkVjZHNhU2VjcDI1NmsxVmVyaWZpY2F0aW9uS2V5MjAxOSIsInB1cnBvc2VzIjpbImF1dGhlbnRpY2F0aW9uIiwiYXNzZXJ0aW9uTWV0aG9kIl19XSwic2VydmljZXMiOlt7ImlkIjoibGlua2VkZG9tYWlucyIsInR5cGUiOiJMaW5rZWREb21haW5zIiwic2VydmljZUVuZHBvaW50Ijp7Im9yaWdpbnMiOlsiaHR0cHM6Ly9hZG1pbi10ZXN0cy1kb21haW4uY29tLyJdfX0seyJpZCI6Imh1YiIsInR5cGUiOiJJZGVudGl0eUh1YiIsInNlcnZpY2VFbmRwb2ludCI6eyJpbnN0YW5jZXMiOlsiaHR0cHM6Ly9kZXYuaHViLm1zaWRlbnRpdHkuY29tL3YxLjAvOWM1OWJlOGItYmQxOC00NWQ5LWI5ZDktMDgyYmMwN2MwOTRmIl19fV19fV19fQ=="), 
    "opIndex" : NumberInt(0), 
    "txnNumber" : NumberLong(9047095205953548), 
    "txnTime" : NumberInt(2106441)
}
{ 
    "_id" : ObjectId("61ae8c5bcfd5dc4f51105fee"), 
    "type" : "create", 
    "didSuffix" : "EiDjX99jWLtd510_Gs8k7wnIcunN9ghbQS3WYWCmwxu2nw", 
    "operationBufferBsonBinary" : BinData(0, "eyJ0eXBlIjoiY3JlYXRlIiwic3VmZml4RGF0YSI6eyJkZWx0YUhhc2giOiJFaUJLWE5LcDZvQ1hlLWM1SENlcHlocGZQb2JEcTVfNXA1QnktOTBpcVhMYWx3IiwicmVjb3ZlcnlDb21taXRtZW50IjoiRWlCU0x6ZUFNd0RUZnRaREtVcGRoYnlWT0ZCWHVIdnlORnVUMnUtcEJtbzFadyJ9LCJkZWx0YSI6eyJ1cGRhdGVDb21taXRtZW50IjoiRWlDLXpUdzRBRjEzbkthRG01MjVZNTVCUktEZUsxMjh0ajF4UUdmbXJBeVJTdyIsInBhdGNoZXMiOlt7ImFjdGlvbiI6InJlcGxhY2UiLCJkb2N1bWVudCI6eyJwdWJsaWNLZXlzIjpbeyJpZCI6InNpZ18wY2UyZWUyZCIsInB1YmxpY0tleUp3ayI6eyJrdHkiOiJFQyIsImNydiI6InNlY3AyNTZrMSIsIngiOiIzeEQwZnhocHB4QkZZejVOUGZMZ3ItNnB1c1NIZWgtN3FQU3FtM3kxdnFzIiwieSI6IjhpaEl6NnZrdlpQSHZkdTFJc09IemJYWHUxV2JmNk1ldWY1SU9rcGxvaU0ifSwidHlwZSI6IkVjZHNhU2VjcDI1NmsxVmVyaWZpY2F0aW9uS2V5MjAxOSIsInB1cnBvc2VzIjpbImF1dGhlbnRpY2F0aW9uIiwiYXNzZXJ0aW9uTWV0aG9kIl19XSwic2VydmljZXMiOlt7ImlkIjoibGlua2VkZG9tYWlucyIsInR5cGUiOiJMaW5rZWREb21haW5zIiwic2VydmljZUVuZHBvaW50Ijp7Im9yaWdpbnMiOlsiaHR0cHM6Ly9hZG1pbi10ZXN0cy1kb21haW4uY29tLyJdfX0seyJpZCI6Imh1YiIsInR5cGUiOiJJZGVudGl0eUh1YiIsInNlcnZpY2VFbmRwb2ludCI6eyJpbnN0YW5jZXMiOlsiaHR0cHM6Ly9kZXYuaHViLm1zaWRlbnRpdHkuY29tL3YxLjAvOWM1OWJlOGItYmQxOC00NWQ5LWI5ZDktMDgyYmMwN2MwOTRmIl19fV19fV19fQ=="), 
    "opIndex" : NumberInt(1), 
    "txnNumber" : NumberLong(9047095205953548), 
    "txnTime" : NumberInt(2106441)
}
{ 
    "_id" : ObjectId("61ae885ecfd5dc4f5110559b"), 
    "type" : "create", 
    "didSuffix" : "EiC9HGkHMUD4UAS-IIPWaVm0EFPW3Ypoybka38jH4H9apg", 
    "operationBufferBsonBinary" : BinData(0, "eyJ0eXBlIjoiY3JlYXRlIiwic3VmZml4RGF0YSI6eyJkZWx0YUhhc2giOiJFaURKVHp6NnRVVzlKZ2l1ZXFvRHc4bkNIazMxSEVOdkJvTUVONDlUYXRzdi13IiwicmVjb3ZlcnlDb21taXRtZW50IjoiRWlEdnNhLWFLQnh1alprTURYdVNUai1kbHQxMWhDajZpX216NnhZNVdVYVVaUSJ9LCJkZWx0YSI6eyJ1cGRhdGVDb21taXRtZW50IjoiRWlDTXo3b2RNUUM2REZ0MzFrMk84RTZvd1dBQTZaZ05kOFdldGh4MnpYQjFIZyIsInBhdGNoZXMiOlt7ImFjdGlvbiI6InJlcGxhY2UiLCJkb2N1bWVudCI6eyJwdWJsaWNLZXlzIjpbeyJpZCI6InNpZ185ZWM2NGEzNiIsInB1YmxpY0tleUp3ayI6eyJrdHkiOiJFQyIsImNydiI6InNlY3AyNTZrMSIsIngiOiIyVExBZk1oWHpvbDlEM0h6ck9NZmY4bm1XV0U3S1hXQ1RIaVp6MjEyekFFIiwieSI6IjVLZTRCRzF4SndITGdYaERwY01Gb2NQOGJENGhGTHhmMElmeHZDUFJHTzQifSwidHlwZSI6IkVjZHNhU2VjcDI1NmsxVmVyaWZpY2F0aW9uS2V5MjAxOSIsInB1cnBvc2VzIjpbImF1dGhlbnRpY2F0aW9uIiwiYXNzZXJ0aW9uTWV0aG9kIl19XSwic2VydmljZXMiOlt7ImlkIjoibGlua2VkZG9tYWlucyIsInR5cGUiOiJMaW5rZWREb21haW5zIiwic2VydmljZUVuZHBvaW50Ijp7Im9yaWdpbnMiOlsiaHR0cHM6Ly9hZG1pbi10ZXN0cy1kb21haW4uY29tLyJdfX0seyJpZCI6Imh1YiIsInR5cGUiOiJJZGVudGl0eUh1YiIsInNlcnZpY2VFbmRwb2ludCI6eyJpbnN0YW5jZXMiOlsiaHR0cHM6Ly9kZXYuaHViLm1zaWRlbnRpdHkuY29tL3YxLjAvOWM1OWJlOGItYmQxOC00NWQ5LWI5ZDktMDgyYmMwN2MwOTRmIl19fV19fV19fQ=="), 
    "opIndex" : NumberInt(0), 
    "txnNumber" : NumberLong(9047086616018952), 
    "txnTime" : NumberInt(2106439)
}

I see other instances of this happening where it's sometimes 3 different Sidetree transactions that get the same number but only one entry appears with however many DID operations went for it. They don't always appear to have the same transaction number if they were in the same block but it's happening frequently enough to cause me issues with my own DID operations.

Sidetree transaction found; adding {"transactionNumber":9047082321051672,"transactionTime":2106438,"transactionTimeHash":"000000008481beb51d1677ca8afafa52eb8d8c18e63b129a0db3099974508d23","anchorString":"2.QmVvVD5VjcAiXBc9uM1h2MmVMYUg1gieAXWCLxX61HXFGe","transactionFeePaid":1362,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}
Sending jRPC request: id: 5gflhh3s3al, method: getrawtransaction
Sending jRPC request: id: 2inhjhmam3a, method: getrawtransaction
Sending jRPC request: id: 4c0spdr34f7, method: getrawtransaction
Sidetree transaction found; adding {"transactionNumber":9047082321051672,"transactionTime":2106438,"transactionTimeHash":"000000008481beb51d1677ca8afafa52eb8d8c18e63b129a0db3099974508d23","anchorString":"1.QmQHj44xFT4uq218DUiFqKGqf6ZYL8Cf2EBQiPyZxsL9XG","transactionFeePaid":1362,"writer":"2a51cd781e8ffcca9f6a216a6d27517654ab4131"}
Sending jRPC request: id: 5c44aead2sl, method: getrawtransaction
Sending jRPC request: id: 44ejukh9cmh, method: getrawtransaction
Sending jRPC request: id: 3b48n8drt68, method: getrawtransaction
Sidetree transaction found; adding {"transactionNumber":9047082321051672,"transactionTime":2106438,"transactionTimeHash":"000000008481beb51d1677ca8afafa52eb8d8c18e63b129a0db3099974508d23","anchorString":"1.QmawBwEpuvg2NQFW8g7HT1iRSH5rDXXsjr4VhmpmYJndik","transactionFeePaid":1362,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}

Any insights? I suppose there could be another reason they aren't resolving but I don't see any indication of problems in the node. It never appears to even try to pull down the IPFS core file from the anchor string of the other transactions - only the one that gets referenced in the mongodb.

tonyimpervious avatar Dec 06 '21 22:12 tonyimpervious

Sounds like a question for @gjgd. I'm guessing there can be multiple outputs per transaction. And Ion checks to see if an output has the type of OP_RETURN and a string values of 'ion:[anchor string]' https://github.com/gjgd/ion-block-explorer/blob/main/watcher/scanBlocks.js#L55c.

If the observer is finding multiple valid strings in a single transaction with a valid anchor string, I'm not sure why that would cause them to be ignored and not get pulled into MongoDB to be resolved.

BenjaminMoe avatar Dec 07 '21 14:12 BenjaminMoe

I think there is a constraint that there can only be 1 Sidetree transaction = 1 anchor file per block. This is because nodes need to agree on the order in which transactions appeared (Although there is a canonical order of transactions within a Bitcoin block as well 🤔)

From the spec

The logical order of operations, as determined by the underlying anchoring system (e.g. Bitcoin block and transaction order). Anchoring systems may widely vary in how they determine the logical order of operations, but the only requirement of an anchoring system is that it can provide a means to deterministically order each operation within a DID’s operational lineage.

gjgd avatar Dec 07 '21 15:12 gjgd

I think there is a constraint that there can only be 1 Sidetree transaction = 1 anchor file per block

Surely that can't be the case. I do see some instances of operations spanning multiple Sidetree transactions in the same block (testnet block 2106158 for instance), but the ION assigned transaction number is different, with different writers even.

It would be impossible to rely upon ION otherwise, which is why I'm just thinking there's some bug here assigning the same number to multiple Sidetree transactions and some overwrite/conflict.

tonyimpervious avatar Dec 07 '21 16:12 tonyimpervious

Maybe @thehenrytsai knows

gjgd avatar Dec 07 '21 16:12 gjgd

Something seems up with the math here: https://github.com/decentralized-identity/sidetree/blob/6a6ef478ada962de9171315e10b4dd2a8deaecfd/lib/bitcoin/TransactionNumber.ts#L12

From this block 2106512, there's 3 ION transactions in a row (52, 53, 54): https://blockstream.info/testnet/nojs/block/0000000000534c2f659635e8abfb3d73f67289bae574239955615005045a878b?start=50

Taking those numbers (minus 1 for counting from 0), you come up with the same exact number: 9047400148631604

Taking this code snippet:

fifty = 2106512 * (2 ** 32) + 50;
console.log("50: " + fifty);

fiftyOne = 2106512 * (2 ** 32) + 51;
console.log("51: " + fiftyOne);

fiftyTwo = 2106512 * (2 ** 32) + 52;
console.log("52: " + fiftyTwo);

fiftyThree = 2106512 * (2 ** 32) + 53;
console.log("53: " + fiftyThree);

fiftyFour = 2106512 * (2 ** 32) + 54;
console.log("54: " + fiftyFour);

fiftyFive = 2106512 * (2 ** 32) + 55;
console.log("55: " + fiftyFive);

Gives you:

50: 9047400148631602
51: 9047400148631604
52: 9047400148631604
53: 9047400148631604
54: 9047400148631606
55: 9047400148631608

Which is what I see when Sidetree is trying to parse the transactions:

Sidetree transaction found; adding {"transactionNumber":9047400148631604,"transactionTime":2106512,"transactionTimeHash":"0000000000534c2f659635e8abfb3d73f67289bae574239955615005045a878b","anchorStrin
g":"2.QmQEuRngGjh9ZeHbWtyeEuccs9TDq6boSVxAavxEwnkjfR","transactionFeePaid":1363,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}                                                                    
Sending jRPC request: id: 7sru4da4cn1, method: getrawtransaction                                                                                                                                        
Sending jRPC request: id: 7dvqmnpb8h9, method: getrawtransaction
Sending jRPC request: id: 4vo9thrhlc5, method: getrawtransaction
Sidetree transaction found; adding {"transactionNumber":9047400148631604,"transactionTime":2106512,"transactionTimeHash":"0000000000534c2f659635e8abfb3d73f67289bae574239955615005045a878b","anchorString":"1.QmemTBmewmoh6pB2M3caZiGzYdQC44wqTyhHZCUNDssGG9","transactionFeePaid":1363,"writer":"2a51cd781e8ffcca9f6a216a6d27517654ab4131"}
Sending jRPC request: id: 1afckb13sfq, method: getrawtransaction
Sending jRPC request: id: ihbhkspukc, method: getrawtransaction
Sending jRPC request: id: 3np239ssl26, method: getrawtransaction
Sidetree transaction found; adding {"transactionNumber":9047400148631604,"transactionTime":2106512,"transactionTimeHash":"0000000000534c2f659635e8abfb3d73f67289bae574239955615005045a878b","anchorString":"2.QmUumGpF21WvxUmJx9vHopUstJhGmNSYnca8j2nsuTH976","transactionFeePaid":1363,"writer":"7628046f6ec479c031d03a47b6a003c4dedb1cad"}
Calculated raw normalized fee for block 2106512: 1349.0342856799323
Finished processing blocks 2106512 to 2106512
Event emitted: bitcoin_observing_loop_success
Transactions request: since transaction number 9047395853664276, time hash '000000000000001dab0b94426e5af717d7c6c718d8f2417c780f96a6a56898db'...
Starting lock resolution for identifier: 

Maybe I should open up this issue with the Sidetree repo? I'm unfamiliar with the particulars of javascript math but this seems to be the issue.

tonyimpervious avatar Dec 07 '21 17:12 tonyimpervious

@impant, wow, will prioritize looking at this as this at a glance looks like a significant bug that would require an immediate hot fix.

thehenrytsai avatar Dec 07 '21 22:12 thehenrytsai

Adding @xinaxu for context.

Just did a bit of research and I can confirm that this is indeed an issue, great find @impant!!

So, the summary:

JavaScript can only store up to 53 bit integer before starting to loose precision.

2 ^ 53 = 9,007,199,254,740,992 = transaction number computed using block height of 2097152.

We happened to recently crossed that block height on bitcoin testnet hence the issue you are seeing. Fortunately we are still long way off from crossing that in bitcoin mainnet, but never the less we need to fix this bug.

We will submit a PR shortly to address this. Thanks again in discovering and debugging this issue!

Thanks, Henry = ]

thehenrytsai avatar Dec 07 '21 22:12 thehenrytsai

Note to implementer:

Investigated. No "quick fix", the fix would involve:

  1. Reimplement the transaction number logic to produce a smaller integer. There is an opportunity here to make the transaction number much more human-friendly also, the current transaction number format had been annoying me for a long time.

    Specifically, the current transaction number is simply a large ugly looking number that doesn't make any sense to a human. It would be much better and easier to make sense of (thus debug), if for example, an ION transaction on index position 500 in block 777777 is 777777000500, as opposed to 3340526778581492. Notice the number is smaller AND easier to make sense of.

  2. Will need to hike DB version.

  3. No "data corruption" concern to ION Core after ION Bitcoin is upgraded but while ION Core is still on an old version, this is because ION Bitcoin will record transactions with much smaller numbers, to which ION Core will think that it is up-to-date since Core would have much larger "transaction-number watermark" saved in its DB.

thehenrytsai avatar Dec 08 '21 02:12 thehenrytsai

Thanks for the investigation!

tonyimpervious avatar Dec 08 '21 14:12 tonyimpervious

Temp solution in the meantime, edit testnet-bitcoin-config.json and set sidetreeTransactionFeeMarkupPercentage to something higher than the default 1, just to move the transaction somewhere else in the block list. I picked 5 and it got it far enough to process correctly.

tonyimpervious avatar Dec 08 '21 16:12 tonyimpervious

Pull request submitted https://github.com/decentralized-identity/sidetree/pull/1167.

Upon real testing, point number 3 above has a caveat, while it is still true that no data corruption will occur if ION Core is kept running on older build, it will however go into a chain reorg infinite loop that does no op. This is bad from perf/stress stand point, so it is best that Core and Bitcoin service are updated to the same new build at the same time while both services are NOT running.

thehenrytsai avatar Jan 06 '22 00:01 thehenrytsai