hedera-sdk-js icon indicating copy to clipboard operation
hedera-sdk-js copied to clipboard
verified publisher icon hashgraph

Increase in returning invalid node account error

Open SimiHunjan opened this issue 1 year ago • 30 comments

Description

This report is from the Kabila team.

  1. I believe since we updated to "@hashgraph/sdk": "2.45.0"
  2. We were using 2.44.0 and before we never had these errors, this change was made 2 weeks ago
  3. The payload is the same than any other transaction, we do not have logs of nodes or something else.
  4. The exact flow is just generate a transaction in a DApp, sign within a wallet and execute in the DApp. It happens super randomly, you can execute 50 transaction and no one of them fails and then execute 3 in a row and these 3 fail.
  5. Browser & Browser + Backend. Browser Flow: DApp create the transaction --> Send to wallet --> Wallet Signs --> Return to DApp to execute. Browser + Backend: Transaction is being created in Backend --> returned to the FrontEnd --> FrontEnd signs --> Send back to the backend to execute. In both flows happen.
  6. The node custom list on Wallet is all the proxies except the 15 (Since 15 is always down, the DNS does not resolve) and from the DApp as any other DApp using wallet connect, you can only generate the transaction with one node, so we choose from the API whenever we setup the client one ramdom node that is available
  7. Mainnet.

The issues are being reported using Kabila Wallet and Hashpack Wallet atm

cc:

Steps to reproduce

See above

Additional context

No response

Hedera network

mainnet

Version

2.45.0

Operating system

None

SimiHunjan avatar Jun 10 '24 17:06 SimiHunjan

Screenshot 2024-06-11 at 13 23 52 Could be related. Happens super rare for me (noticed this 2 times for last week).

moshenskyDV avatar Jun 11 '24 12:06 moshenskyDV

+1 We have also been seeing elevated reports of this error, though from what I've seen it may not be SDK related

teacoat avatar Jun 12 '24 18:06 teacoat

Noting that SaucerSwap is using v2.30.0 and Kabila/HashPack are on v2.45.0 and both experienced an uptick of seeing this error over the last week or so.

SimiHunjan avatar Jun 12 '24 20:06 SimiHunjan

Do we have any idea what's causing this error? We keep seeing more and more often... getting daily reports from users.

AdrianKBL avatar Jun 13 '24 11:06 AdrianKBL

@AdrianKBL we are still trying to reproduce it and communicating with our colleagues. Did you noticed any specific transaction fail more often? You mentioned that it happens randomly but I am wondering if there is any specific transaction that is executed very often and started to fail more?

agadzhalov avatar Jun 13 '24 12:06 agadzhalov

Hello @agadzhalov,

I don't think It happens in any specific transaction more often. We've encountered this error across a wide range of transactions, including sending Hbar, transferring HTS tokens, associating tokens, granting allowances, and executing purchases in the Marketplace, among others.

All the transactions generate via DApp (FrontEnd) --> Sending to Wallet and Signing (Front End) --> Returning back to DApp and Executing (FrontEnd).

AdrianKBL avatar Jun 13 '24 13:06 AdrianKBL

@AdrianKBL is it possible for your team to try and enable getting logs for future failure reports? If we had the raw transaction the users signed and following which node the transaction was sent to that provided the error that would help getting to the root of this.

To confirm, the error being received is INVALID_NODE_ACCOUNT and not INVALID_NODE_ ACCOUNT_ID @teacoat @AdrianKBL

SimiHunjan avatar Jun 13 '24 16:06 SimiHunjan

Description of the error:

Node Account provided does not match the node account of the node the transaction was submitted to.

https://github.com/hashgraph/hedera-protobufs/blob/main/services/response_code.proto#L48

SimiHunjan avatar Jun 13 '24 16:06 SimiHunjan

@AdrianKBL in what step of your flow does the node account ID get set on the transaction? Does the wallet set that parameter for the transaction or the Dapp in the first step?

SimiHunjan avatar Jun 13 '24 16:06 SimiHunjan

The DApp set the node as it requires the WalletConnect protocol.

AdrianKBL avatar Jun 13 '24 16:06 AdrianKBL

Feedback From SaucerSwap:

sslittletarzan 8:46 AM

  • Hello, I haven't had this issue in the last 6 days or so but I will do my best:
  • About 10 days ago
  • No, 2.30.0
  • Smart contract calls, most often failing when I use await Promise.all for them in typescript. The transaction failed precheck with INVALID_NODE_ACCOUNT
  • It was normal contract calls that failed, nothing too crazy. For some scripts I call 20-30 contracts in a batch using await Promise.all
  • They are typescript scripts using only the sdk
  • Yes, the default node selection feature
  • Mainnet

SimiHunjan avatar Jun 14 '24 17:06 SimiHunjan

@teacoat @AdrianKBL can you provide us a link from hashscan from such a transaction?

agadzhalov avatar Jun 18 '24 10:06 agadzhalov

@agadzhalov The transaction doesn't even hit consensus, what fails is the PRE_CHECK , so there aren't any transaction failed on HashScan.

AdrianKBL avatar Jun 18 '24 10:06 AdrianKBL

@AdrianKBL this fails with INVALID_NODE_ACCOUNT?

agadzhalov avatar Jun 18 '24 10:06 agadzhalov

Edited my first comment and yes fails with INVALID_NODE_ACCOUNT @agadzhalov

AdrianKBL avatar Jun 18 '24 10:06 AdrianKBL

We are also seeing this quite regularly and intermittently when we call for the record on a transaction response:

const transactionResponse = await transferTransaction.execute(client);
const transactionRecord = await transactionResponse.getRecord(client);

{
  "name": "StatusError",
  "status": "INVALID_NODE_ACCOUNT",
  "transactionId": "[email protected]",
  "message": "transaction [email protected] failed precheck with status INVALID_NODE_ACCOUNT",
  "contractFunctionResult": null
}

justynspooner avatar Jul 15 '24 13:07 justynspooner

@justynspooner what environment is the transaction being executed in? (node.js or browser/web app)

SimiHunjan avatar Jul 15 '24 16:07 SimiHunjan

@justynspooner what environment is the transaction being executed in? (node.js or browser/web app)

It's being run in a Node environment.

justynspooner avatar Jul 17 '24 11:07 justynspooner

@justynspooner so it's a backend?

agadzhalov avatar Jul 17 '24 11:07 agadzhalov

I also get INVALID_NODE_ACCOUNT from time to time.

In an attempt to try and narrow down the node ID, I fetched all IP addresses from https://mainnet-public.mirrornode.hedera.com/api/v1/network/nodes

Example:

{"ip_address_v4":"34.239.82.6","port":50211} and {"ip_address_v4":"35.237.200.180","port":50211} for node id 0 (LG)

Then I used Client.forNetwork() instead of Client.forMainnet() to manually set the network (one IP at a time). The test is simple - sending a transaction to the consensus node and see if it works or not.

The Tata Communications node (node id 10, account id: 0.0.13) - 35.204.86.32:50211 and 35.177.162.180:50211 both give me INVALID_NODE_ACCOUNT and it is reproducible.

Also the Zain node (node id 6, account id: 0.0.9) - 35.197.192.225:50211 and 35.181.158.250:50211 hangs when I try and send a transaction to it. That one seems problematic too.

Node 0, 1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14 works fine so far. [Update] - also checked 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 and 31 - they all worked.

So the only two problematic ones are: 10 (INVALID_NODE_ACCOUNT) and 6 (Tnx just hangs with no response)

Perhaps the Tata Communications node is the problematic node causing those random INVALID_NODE_ACCOUNT when using Client.ForMainnet() ? Just a guess.

iron4548 avatar Jul 20 '24 10:07 iron4548

@justynspooner so it's a backend?

Yes, a backend server.

justynspooner avatar Jul 20 '24 22:07 justynspooner

Hi @iron4548 and thank you for your response.

I guess you are aware we have 31 nodes (account 0.0.3 - account 0.0.34). The page you’ve pointed https://mainnet-public.mirrornode.hedera.com/api/v1/network/nodes contains only the first 9 nodes. It’s pretty possible you’ve noticed mainnet-public.mirrornode.hedera.com uses pagination and there are also three other pages of nodes - one, two, three.

Node 10 (Account id: 0.0.13) which you are referring to the node we can use to reproduce the INVALID_NODE_ACCOUNT error can be found on the second page. The IPs that you’ve provided for Node 10 (0.0.13) are incorrect, those are actually the IPs of Node 9 (0.0.12). In this scenario is expected to receive INVALID_NODE_ACCOUNT as the IPs are wrong.

Node 9 (0.0.12) DLA Piper

35.204.86.32:50211
35.177.162.180:50211
35.204.86.32:50212
35.177.162.180:50212

Node 10 (0.0.13) Tata Communications

34.215.192.104:50211
35.234.132.107:50211
34.215.192.104:50212
35.234.132.107:50212

For Node 6 (0.0.9) you can check the status in Hedera Status that it actually says Partial Outage. In this scenario when you try to execute a transaction against this node you’ll receive Received RST_STREAM with code 0 which is different than INVALID_NODE_ACCOUNT error and it’s kind of expected since the status of the node is partial outage.

I am providing a snippet demonstrating that Node 10 (0.0.13) actually works when configured with its corresponding IPs. Each 10 seconds a transfer transaction is executed on mainnet.

import {
    AccountId,
    PrivateKey,
    Client,
    Hbar,
    TransferTransaction,
    Logger,
    LogLevel,
} from "@hashgraph/sdk";

import dotenv from "dotenv";

dotenv.config();

async function main() {
 
    if (
        process.env.OPERATOR_ID == null ||
        process.env.OPERATOR_KEY == null ||
        process.env.HEDERA_NETWORK == null
    ) {
        throw new Error(
            "Environment variables OPERATOR_ID, HEDERA_NETWORK, and OPERATOR_KEY are required.",
        );
    }
    // Configure accounts and client, and generate needed keys
    const operatorId = AccountId.fromString(process.env.OPERATOR_ID);
    //const operatorKey = PrivateKey.fromStringDer(process.env.OPERATOR_KEY);
    const operatorKey = PrivateKey.fromStringECDSA(process.env.OPERATOR_KEY)

    let attempt = 0;

    setInterval(async () => {
        try {
            console.log("=====================" + attempt + "========================");
            
            const nodes = {
                '34.215.192.104:50211': new AccountId(13),
                '35.234.132.107:50211': new AccountId(13),
            }

            const client = Client.forNetwork(nodes);
            client.setOperator(operatorId, operatorKey);

            const infoLogger = new Logger(LogLevel.Info);
            client.setLogger(infoLogger);

            const newAccountId = new AccountId(6190496);

            const sendHbar = await new TransferTransaction()
                .addHbarTransfer(operatorId, Hbar.fromTinybars(-1)) //Sending account
                .addHbarTransfer(newAccountId, Hbar.fromTinybars(1)) //Receiving account
                .execute(client);

            const transactionReceipt = await sendHbar.getReceipt(client);
            console.log("The transfer transaction from account " + operatorId + " to the new account " + newAccountId  + " was: " + transactionReceipt.status.toString());

            console.log("==============================================");
        } catch (e) {
            console.log(e);
        }
        attempt++;
    }, 10000)

}

void main();

agadzhalov avatar Jul 22 '24 12:07 agadzhalov

Hi @iron4548 and thank you for your response.

I guess you are aware we have 31 nodes (account 0.0.3 - account 0.0.34). The page you’ve pointed https://mainnet-public.mirrornode.hedera.com/api/v1/network/nodes contains only the first 9 nodes. It’s pretty possible you’ve noticed mainnet-public.mirrornode.hedera.com uses pagination and there are also three other pages of nodes - one, two, three.

Node 10 (Account id: 0.0.13) which you are referring to the node we can use to reproduce the INVALID_NODE_ACCOUNT error can be found on the second page. The IPs that you’ve provided for Node 10 (0.0.13) are incorrect, those are actually the IPs of Node 9 (0.0.12). In this scenario is expected to receive INVALID_NODE_ACCOUNT as the IPs are wrong.

Node 9 (0.0.12) DLA Piper

35.204.86.32:50211
35.177.162.180:50211
35.204.86.32:50212
35.177.162.180:50212

Node 10 (0.0.13) Tata Communications

34.215.192.104:50211
35.234.132.107:50211
34.215.192.104:50212
35.234.132.107:50212

For Node 6 (0.0.9) you can check the status in Hedera Status that it actually says Partial Outage. In this scenario when you try to execute a transaction against this node you’ll receive Received RST_STREAM with code 0 which is different than INVALID_NODE_ACCOUNT error and it’s kind of expected since the status of the node is partial outage.

I am providing a snippet demonstrating that Node 10 (0.0.13) actually works when configured with its corresponding IPs. Each 10 seconds a transfer transaction is executed on mainnet.

You're right. Somehow I got the IP mixed up in the process. I should have double checked the source. Apologies for that.

A question - for Client.ForMainnet() - where is it getting the list of node addresses & ip from? Is it hard coded somewhere? I'm wondering if one of the node in the address book have incorrect IP causing INVALID_NODE_ACCOUNT. Just a thought.

Also INVALID_NODE_ACCOUNT for incorrect IP address is a bit misleading. That error code implies there's something wrong with 0.0.13 (the account). INVALID_NODE_ACCOUNT_IP would have been more clear?

iron4548 avatar Jul 22 '24 21:07 iron4548

@iron4548

Client.ForMainnet() is getting the list of IPs from the Address Book and more specifically from mainnet-public.mirrornode.hedera.com:443. You can read more about in the docs. In the code you can see it here.

We've tested against each individual IP and each individual node and we didn't manage to reproduce the error. All of the nodes and the IPs in the address books are correct.

agadzhalov avatar Jul 23 '24 08:07 agadzhalov

Hi guys, we've seen the INVALID_NODE_ACCOUNT error appearing in Guardian production environments (nodejs backends) with version 2.46. I don't think this is related to the address book, it seems the SDK simply sometimes mixes the tx prepared for a specific node and it sends that to a different node.

Neurone avatar Nov 18 '24 13:11 Neurone

PR: https://github.com/hashgraph/solo/pull/840 Example instance: https://github.com/hashgraph/solo/actions/runs/11895809133/job/33148579450?pr=840

1) E2E Test Suite for 'node-update'
       node-update
         Node update
           should update a new node property successfully:
     SoloError: Error in updating nodes: Error in prepare upgrade: transfer amount failed with an error: {"name":"StatusError","status":"INVALID_NODE_ACCOUNT","transactionId":"[email protected]","message":"receipt for transaction [email protected] contained error status INVALID_NODE_ACCOUNT"}
      at file:///home/runner/_work/solo/solo/src/core/helpers.ts:401:19
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async NodeCommandHandlers.update (file:///home/runner/_work/solo/solo/src/commands/node/handlers.ts:247:9)
      at async Context.<anonymous> (file:///home/runner/_work/solo/solo/test/e2e/commands/node_update.test.ts:77:13)
     Caused by: SoloError: Error in prepare upgrade: transfer amount failed with an error: {"name":"StatusError","status":"INVALID_NODE_ACCOUNT","transactionId":"0.0.2@17[319](https://github.com/hashgraph/solo/actions/runs/11895809133/job/33148579450?pr=840#step:15:320)46760.855168555","message":"receipt for transaction [email protected] contained error status INVALID_NODE_ACCOUNT"}
      at Task.taskFn (file:///home/runner/_work/solo/solo/src/commands/node/tasks.ts:424:23)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Task.run (file:///home/runner/_work/solo/solo/node_modules/listr2/dist/index.js:2058:11)
     Caused by: SoloError: transfer amount failed with an error: {"name":"StatusError","status":"INVALID_NODE_ACCOUNT","transactionId":"[email protected]","message":"receipt for transaction [email protected] contained error status INVALID_NODE_ACCOUNT"}
      at AccountManager.transferAmount (file:///home/runner/_work/solo/solo/src/core/account_manager.ts:536:19)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Task.taskFn (file:///home/runner/_work/solo/solo/src/commands/node/tasks.ts:405:17)
      at async Task.run (file:///home/runner/_work/solo/solo/node_modules/listr2/dist/index.js:2058:11)
     Caused by: StatusError: receipt for transaction [email protected] contained error status INVALID_NODE_ACCOUNT
      at new ReceiptStatusError (file:///home/runner/_work/solo/solo/node_modules/@hashgraph/sdk/src/ReceiptStatusError.js:37:9)
      at TransactionReceiptQuery._mapStatusError (file:///home/runner/_work/solo/solo/node_modules/@hashgraph/sdk/src/transaction/TransactionReceiptQuery.js:336:16)
      at TransactionReceiptQuery.execute (file:///home/runner/_work/solo/solo/node_modules/@hashgraph/sdk/src/Executable.js:730:32)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async TransactionResponse.getReceipt (file:///home/runner/_work/solo/solo/node_modules/@hashgraph/sdk/src/transaction/TransactionResponse.js:81:25)
      at async AccountManager.transferAmount (file:///home/runner/_work/solo/solo/src/core/account_manager.ts:529:29)
      at async Task.taskFn (file:///home/runner/_work/solo/solo/src/commands/node/tasks.ts:405:17)
      at async Task.run (file:///home/runner/_work/solo/solo/node_modules/listr2/dist/index.js:2058:11)

logs attached. Rerunning the job passed. hashgraph-sdk.log solo.log

jeromy-cannon avatar Nov 18 '24 18:11 jeromy-cannon

Hi guys, I was able to reproduce the issue in a common scenario, I assume this may be the source for the others too, but not 100% sure. I have been testing the latest v2.54.0-beta.1.

I created two clients with different node configurations. The first client sends the tx, the second asks for the receipt and the record data. My goal was to simulate the SDK changing node on the fly via code or for an overlapping thread.

Essentially, the client is not 100% thread safe and I see different behaviour when I get the receipts/record from the response object after sending a tx or when I use the txID.

Using the txID is the safest approach and the client seems to work cleanly.

When using the response object, the client mixes up the nodes, and in my example where the nodes don't overlap, it always ends up with an INVALID_NODE_ACCOUNT error.

See for details the log when I ask for the receipt or the record data via the tx response.

The only doubt I still have is that while the bug is always reproducible, was only able to get the error when querying for the record and not for the receipt. My guess this is because getting a receipt is a free operation, while getting the record is a costly operation, so the Consensus Node ignores the node accountID field, but this is a feeling, I didn't verify it.

Regardless, once we solve this, I think we will most likely solve the other strange cases as well.

Below is the code and the log.

require("dotenv").config();
const {
  TransferTransaction,
  Hbar,
  AccountId,
  Client,
  TransactionReceiptQuery,
  TransactionRecordQuery,
} = require("@hashgraph/sdk");

const myAccountId = process.env.OPERATOR_ID;
const myPrivateKey = process.env.OPERATOR_KEY;

async function main() {
  console.log("=======", "USING NODE 0.0.7");
  let client = Client.forNetwork({
    "34.94.160.4:50211": new AccountId(7),
  });
  client.setOperator(myAccountId, myPrivateKey);
  client.setLogger(console);

  let txResponse = await new TransferTransaction()
    .addHbarTransfer(client.getOperator().accountId, Hbar.fromTinybars(-1))
    .addHbarTransfer("0.0.2", Hbar.fromTinybars(1))
    .execute(client);

  let txId = txResponse.transactionId;
  console.log("=======", "TXID", txId.toString());

  console.log("=======", "CHANGING TO NODE 0.0.4");
  let newClient = Client.forNetwork({
    "35.237.119.55:50211": new AccountId(4),
  });
  newClient.setOperator(myAccountId, myPrivateKey);
  newClient.setLogger(console);

  console.log("=======", "ASK FOR RECEIPT VIA TX ID");
  let receipt = await new TransactionReceiptQuery()
    .setTransactionId(txId)
    .execute(newClient);
  console.log("=======", "STATUS", receipt.status);

  console.log();

  console.log("=======", "ASK FOR RECEIPT VIA TX RESPONSE");
  receipt = await txResponse.getReceipt(newClient);
  console.log("=======", "STATUS", receipt.status);

  console.log();

  console.log("=======", "ASK FOR RECORD VIA TX ID");
  let record = await new TransactionRecordQuery()
    .setTransactionId(txId)
    .execute(newClient);
  console.log("=======", "STATUS", record.receipt.status);

  console.log();

  console.log("=======", "ASK FOR RECORD VIA TX RESPONSE OBJECT");
  record = await txResponse.getRecord(newClient);
  console.log("=======", "STATUS", record.receipt.status);

  process.exit(0);
}

void main();

❯ node index.js
======= USING NODE 0.0.7
Network used: null
Transaction Body: {"cryptoTransfer":{"transfers":{"accountAmounts":[{"accountID":{"alias":null,"accountNum":{"low":2,"high":0,"unsigned":false},"shardNum":{"low":0,"high":0,"unsigned":false},"realmNum":{"low":0,"high":0,"unsigned":false}},"amount":{"low":1,"high":0,"unsigned":false},"isApproval":false},{"accountID":{"alias":null,"accountNum":{"low":1441,"high":0,"unsigned":false},"shardNum":{"low":0,"high":0,"unsigned":false},"realmNum":{"low":0,"high":0,"unsigned":false}},"amount":{"low":-1,"high":-1,"unsigned":false},"isApproval":false}]},"tokenTransfers":[]},"transactionFee":{"low":100000000,"high":0,"unsigned":false},"memo":"","transactionID":{"accountID":{"alias":null,"accountNum":{"low":1441,"high":0,"unsigned":false},"shardNum":{"low":0,"high":0,"unsigned":false},"realmNum":{"low":0,"high":0,"unsigned":false}},"transactionValidStart":{"seconds":{"low":1731976457,"high":0,"unsigned":false},"nanos":410189574},"scheduled":false,"nonce":null},"nodeAccountID":{"alias":null,"accountNum":{"low":7,"high":0,"unsigned":false},"shardNum":{"low":0,"high":0,"unsigned":false},"realmNum":{"low":0,"high":0,"unsigned":false}},"transactionValidDuration":{"seconds":{"low":120,"high":0,"unsigned":false}}}
[TransferTransaction:1731976457.410189574] Node AccountID: 0.0.7, IP: 34.94.160.4:50211
Trace: [TransferTransaction:1731976457.410189574] sending protobuf 2ab9010a4f0a190a0c0889baefb9061086fecbc30112070800100018a10b180012060800100018071880c2d72f220208783200721f0a1d0a0c0a06080010001802100218000a0d0a070800100018a10b1001180012660a640a2097697c9a01c0274b049625fb6860a9acb2aaeb1cce5d04aeeb9fe5121131dd7a1a40de8d15e5e270c390894d8865d94b6b14226cc4f3b5e4e3ea058db16b6a1bbc3a29d24de51fa4102ca535107e757087d80b41b7fb94938c6b792da26020c1bb02
    at TransferTransaction.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:22:20)
Trace: [TransferTransaction:1731976457.410189574] sending protobuf
    at TransferTransaction.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:22:20)
[TransferTransaction:1731976457.410189574] received status OK
SDK Transaction Status Response: OK
Transaction Info: {"nodeId":"0.0.7","transactionHash":"c37c123958b3c6816ebbcee9e898e2eb600e5b4d1fc1b3775052eb901a440d8fe51e8d866c83498b37e35e686851d4e1","transactionId":"[email protected]"}
======= TXID [email protected]
======= CHANGING TO NODE 0.0.4
======= ASK FOR RECEIPT VIA TX ID
[TransactionReceiptQuery:1731976467035] making a payment transaction for node 0.0.4 and transaction ID [email protected] with amount 0 tℏ
[TransactionReceiptQuery:1731976467035] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionReceiptQuery:1731976467035] making a payment transaction for node 0.0.4 and transaction ID [email protected] with amount 0 tℏ
Trace: [TransactionReceiptQuery:1731976467035] sending protobuf 726d0a500a4c2a4a0a480a190a0c0886baefb90610b99fd0e30212070800100018a10b180012060800100018041880c2d72f22020878721a0a180a0a0a0608001000180010000a0a0a060800100018041000100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:38:17)
Trace: [TransactionReceiptQuery:1731976467035] sending protobuf 72060a0012020815
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:38:17)
[TransactionReceiptQuery:1731976467035] received node precheck status OK
[TransactionReceiptQuery:1731976467035] received receipt status UNKNOWN
[TransactionReceiptQuery:1731976467035] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionReceiptQuery:1731976467035] making a payment transaction for node 0.0.4 and transaction ID [email protected] with amount 0 tℏ
Trace: [TransactionReceiptQuery:1731976467035] sending protobuf 726d0a500a4c2a4a0a480a190a0c088abaefb90610e693949f0212070800100018a10b180012060800100018041880c2d72f22020878721a0a180a0a0a0608001000180010000a0a0a060800100018041000100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:38:17)
Trace: [TransactionReceiptQuery:1731976467035] sending protobuf 722c0a00122808162a240a1008b0ea01109dc9141a060890c6efb906121008b0ea0110c4db191a0608a0e2efb906
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:38:17)
[TransactionReceiptQuery:1731976467035] received node precheck status OK
[TransactionReceiptQuery:1731976467035] received receipt status SUCCESS
======= STATUS Status { _code: 22 }

======= ASK FOR RECEIPT VIA TX RESPONSE
[TransactionReceiptQuery:1731976467996] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 0 tℏ
[TransactionReceiptQuery:1731976467996] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionReceiptQuery:1731976467996] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 0 tℏ
Trace: [TransactionReceiptQuery:1731976467996] sending protobuf 726d0a500a4c2a4a0a480a190a0c0889baefb90610cc9dc0830312070800100018a10b180012060800100018071880c2d72f22020878721a0a180a0a0a0608001000180010000a0a0a060800100018071000100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionResponse.getReceipt (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:88:21)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:46:13)
Trace: [TransactionReceiptQuery:1731976467996] sending protobuf 722c0a00122808162a240a1008b0ea01109dc9141a060890c6efb906121008b0ea0110c4db191a0608a0e2efb906
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionResponse.getReceipt (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:88:21)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:46:13)
[TransactionReceiptQuery:1731976467996] received node precheck status OK
[TransactionReceiptQuery:1731976467996] received receipt status SUCCESS
======= STATUS Status { _code: 22 }

======= ASK FOR RECORD VIA TX ID
[CostQuery:TransactionRecordQuery:1731976456.157796478] making a payment transaction for node 0.0.0 and transaction ID [email protected] with amount 0 tℏ
[CostQuery:TransactionRecordQuery:1731976456.157796478] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
Trace: [CostQuery:TransactionRecordQuery:1731976456.157796478] sending protobuf 7ad8010aba010ab5012ab2010a480a180a0c088abaefb90610abd4819d021206080010001800180012060800100018001880c2d72f22020878721b0a190a0b0a070800100018a10b10000a0a0a06080010001800100012660a640a2097697c9a01c0274b049625fb6860a9acb2aaeb1cce5d04aeeb9fe5121131dd7a1a404af063685272a1a44813d80eec378a2479fbc4d21965c24fe6be715ddeb37bb519bc29bd80bc1693f7fb26f5dd2365e2d4e151964ab76391371e00cf2a597c0f100212190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at CostQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionRecordQuery.getCost (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:185:18)
    at async TransactionRecordQuery._beforeExecute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:295:26)
    at async TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:529:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:52:16)
Trace: [CostQuery:TransactionRecordQuery:1731976456.157796478] sending protobuf 7a060a0410021807
    at CostQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionRecordQuery.getCost (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:185:18)
    at async TransactionRecordQuery._beforeExecute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:295:26)
    at async TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:529:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:52:16)
[TransactionRecordQuery:1731976456.157796478] received node precheck status OK
[TransactionRecordQuery:1731976456.157796478] received cost for query 8 tℏ
[TransactionRecordQuery:1731976456.157796478] making a payment transaction for node 0.0.4 and transaction ID [email protected] with amount 8 tℏ
[TransactionRecordQuery:1731976456.157796478] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionRecordQuery:1731976456.157796478] making a payment transaction for node 0.0.4 and transaction ID [email protected] with amount 8 tℏ
Trace: [TransactionRecordQuery:1731976456.157796478] sending protobuf 7ad9010abb010ab6012ab3010a490a190a0c0888baefb90610bec09e830112070800100018a10b180012060800100018041880c2d72f22020878721b0a190a0b0a070800100018a10b100f0a0a0a06080010001804101012660a640a2097697c9a01c0274b049625fb6860a9acb2aaeb1cce5d04aeeb9fe5121131dd7a1a408e748609561193e596553407c55670eb28f445c58a42f812596462b3de3a748d00f53c40d27aadf82038b59259308884ceb7e4c7ffd30e58a5731ef7b7f4be0a100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:52:16)
Trace: [TransactionRecordQuery:1731976456.157796478] sending protobuf 7aba010a001ab5010a2808162a240a1008b0ea01109dc9141a060890c6efb906121008b0ea0110c4db191a0608a0e2efb9061230c37c123958b3c6816ebbcee9e898e2eb600e5b4d1fc1b3775052eb901a440d8fe51e8d866c83498b37e35e686851d4e11a0b0893baefb90610b8a4e02422130a0c0889baefb9061086fecbc301120318a10b30eabc0552310a060a02180210020a070a02180710e2380a080a02186210b4ba090a090a0318a00610be86010a090a0318a10b10d5f90a
    at TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:52:16)
[TransactionRecordQuery:1731976456.157796478] received node precheck status OK
[TransactionRecordQuery:1731976456.157796478] received record's receipt SUCCESS
======= STATUS Status { _code: 22 }

======= ASK FOR RECORD VIA TX RESPONSE OBJECT
[TransactionReceiptQuery:1731976469096] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 0 tℏ
[TransactionReceiptQuery:1731976469096] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionReceiptQuery:1731976469096] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 0 tℏ
Trace: [TransactionReceiptQuery:1731976469096] sending protobuf 726d0a500a4c2a4a0a480a190a0c088bbaefb906108df1e2b40112070800100018a10b180012060800100018071880c2d72f22020878721a0a180a0a0a0608001000180010000a0a0a060800100018071000100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionResponse.getReceipt (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:88:21)
    at async TransactionResponse.getRecord (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:106:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
Trace: [TransactionReceiptQuery:1731976469096] sending protobuf 722c0a00122808162a240a1008b0ea01109dc9141a060890c6efb906121008b0ea0110c4db191a0608a0e2efb906
    at TransactionReceiptQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionResponse.getReceipt (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:88:21)
    at async TransactionResponse.getRecord (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionResponse.cjs:106:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
[TransactionReceiptQuery:1731976469096] received node precheck status OK
[TransactionReceiptQuery:1731976469096] received receipt status SUCCESS
[CostQuery:TransactionRecordQuery:1731976458.412116987] making a payment transaction for node 0.0.0 and transaction ID [email protected] with amount 0 tℏ
[CostQuery:TransactionRecordQuery:1731976458.412116987] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
Trace: [CostQuery:TransactionRecordQuery:1731976458.412116987] sending protobuf 7ad8010aba010ab5012ab2010a480a180a0c0889baefb906109095edaf021206080010001800180012060800100018001880c2d72f22020878721b0a190a0b0a070800100018a10b10000a0a0a06080010001800100012660a640a2097697c9a01c0274b049625fb6860a9acb2aaeb1cce5d04aeeb9fe5121131dd7a1a4019f35f06fafdcb86a19b1138314f242b8388dfbe137cbfd585274421e20c4706d40c0b728f846c48f6c7440db0d7a291e44ba5663e9e7502a57ca38229e62f0e100212190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at CostQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionRecordQuery.getCost (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:185:18)
    at async TransactionRecordQuery._beforeExecute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:295:26)
    at async TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:529:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
Trace: [CostQuery:TransactionRecordQuery:1731976458.412116987] sending protobuf 7a060a0410021807
    at CostQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async TransactionRecordQuery.getCost (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:185:18)
    at async TransactionRecordQuery._beforeExecute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/query/Query.cjs:295:26)
    at async TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:529:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
[TransactionRecordQuery:1731976458.412116987] received node precheck status OK
[TransactionRecordQuery:1731976458.412116987] received cost for query 8 tℏ
[TransactionRecordQuery:1731976458.412116987] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 8 tℏ
[TransactionRecordQuery:1731976458.412116987] Node AccountID: 0.0.4, IP: 35.237.119.55:50211
[TransactionRecordQuery:1731976458.412116987] making a payment transaction for node 0.0.7 and transaction ID [email protected] with amount 8 tℏ
Trace: [TransactionRecordQuery:1731976458.412116987] sending protobuf 7ad8010aba010ab5012ab2010a480a180a0b088cbaefb906109fa0ff5c12070800100018a10b180012060800100018071880c2d72f22020878721b0a190a0b0a070800100018a10b100f0a0a0a06080010001807101012660a640a2097697c9a01c0274b049625fb6860a9acb2aaeb1cce5d04aeeb9fe5121131dd7a1a40b136831a1f62181d0936a9578c6d4c16b8be4b8a5947c96945683a2677f92fe8c37317b68ba7b9bcb36aaca9b98ddc055d87b2c8f37c96d10476cfa4c03c590c100012190a0c0889baefb9061086fecbc30112070800100018a10b1800
    at TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:616:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
Trace: [TransactionRecordQuery:1731976458.412116987] sending protobuf 7a040a020803
    at TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:644:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12)
[TransactionRecordQuery:1731976458.412116987] received node precheck status INVALID_NODE_ACCOUNT
/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/PrecheckStatusError.cjs:55
    super(props, `transaction ${props.transactionId.toString()} failed precheck with status ${props.status.toString()} against node account id ${props.nodeId.toString()}`);
    ^

PrecheckStatusError: transaction [email protected] failed precheck with status INVALID_NODE_ACCOUNT against node account id 0.0.7
    at new PrecheckStatusError (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/PrecheckStatusError.cjs:55:5)
    at TransactionRecordQuery._mapStatusError (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/transaction/TransactionRecordQuery.cjs:264:16)
    at TransactionRecordQuery.execute (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/node_modules/@hashgraph/sdk/lib/Executable.cjs:668:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/Users/giuseppebertone/workspace/hedera/triage/invalid-account-node/index.js:60:12) {
  status: Status { _code: 3 },
  transactionId: TransactionId {
    accountId: AccountId {
      shard: Long { low: 0, high: 0, unsigned: false },
      realm: Long { low: 0, high: 0, unsigned: false },
      num: Long { low: 1441, high: 0, unsigned: false },
      aliasKey: null,
      evmAddress: null,
      _checksum: null
    },
    validStart: Timestamp {
      seconds: Long { low: 1731976458, high: 0, unsigned: false },
      nanos: Long { low: 412116987, high: 0, unsigned: false }
    },
    scheduled: false,
    nonce: null
  },
  contractFunctionResult: null,
  nodeId: AccountId {
    shard: Long { low: 0, high: 0, unsigned: false },
    realm: Long { low: 0, high: 0, unsigned: false },
    num: Long { low: 7, high: 0, unsigned: false },
    aliasKey: null,
    evmAddress: null,
    _checksum: null
  }
}

Node.js v20.12.2

Neurone avatar Nov 19 '24 00:11 Neurone

After debugging with the team it turns out if a transaction is built to be a submitted to a specific node, but that node is not listed in the Clinet's list instead of throwing an error the SDK will send the transaction to a random node and it will reach the consensus node where it will fail at PRECHECK.

The solution would be to introduce a new error to the user indicating that they are trying to submit a transaction to a node id which is not available in the Client's list and the user will have to adjust its setup. This way the user experience when using the JS SDK and interacting with the Hedera network will be improved.

We've also created an issue for this, the PR is ready and soon new version with these changes will be released.

agadzhalov avatar Dec 06 '24 07:12 agadzhalov

Hi @agadzhalov, thanks for the analysis and explanation.

Skipping the details of why this problem happens from time to time, my personal opinion is that changing the error code is not a final solution.

The current error is clear (and correct), so the problem is not a misunderstanding of the error code or description by the developers.

The problem exists for clients that use a default SDK setup without changing anything in the configuration, so if the client's node list changes, it's not because the developers changed it (at least not voluntarily).

What we want to achieve, I suppose, is that the SDK should never send a transaction signed for one node to another node.

Neurone avatar Dec 06 '24 13:12 Neurone

Actually there are some projects that create a custom node list and remove nodes from their node list that are down as communicated on status page. If the dapp client is setting the node account ID and the wallet client is removing nodes from its list as it learns nodes are down from status page then there will be an inconsistency between what the dapp provided to the wallet and list of nodes in the wallet account.

Agreed that the root of this investigation identified that the SDK would randomly pick a node to submit to if it did not have the node defined in the client that is submitting the transactions. The SDK should not randomly pick a node to send a transaction to and have it error at the consensus node. It should let the user know they are trying to submit a transaction to a node that is node in their client list and try another another node.

SimiHunjan avatar Dec 06 '24 15:12 SimiHunjan