trueblocks-core chifra export: Intrablock transactions for same erc20 tokens and address are collapsed into one transaction.

When you are extracting statements for an address that inside same block has multiple transactions that move the same ERC20 token, then the overall net value of all transfers in the block for that token will be associated with the first transaction in the block. The other transactions will not be shown at all.

For example: In block 14933657 the address: 0x6aED588ca2052CCFC907Db8c24dF4B7B95A29A5E peforms 2 transfers of YFI token (0x0bc529c00C6401aEF6D220BE8C6Ea1667F6Ad93e) to two different recipients:

The first one in the block: txIndex: 303 txHash: 0x458ae31579274588f04850853ec71f67e1e21b2d4e72ada18fdc04352fc22295 to: 0xACA096d2E6492D7380c5d8E70D90E13EEcC72dd4 value: 304442430000000000
The second one in the block: txIndex: 305 txHash: 0x8c1703f5dee17242784c912f208f55508cf003f2b3b0ca9070d9556a619df186 to: 0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d value: 961090000000000

When calling: chifra export --fmt json --accounting --articulate 0x6aED588ca2052CCFC907Db8c24dF4B7B95A29A5E

You can see that under the first transaction in block (with index 303) the following statement is generated for the YFI token:

{
      "blockNumber": 14933657,
      "transactionIndex": 303,
      "timestamp": 1654793734,
      "assetAddr": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
      "assetSymbol": "YFI",
      "decimals": 18,
      "prevBlk": 14927653,
      "prevBlkBal": "18255476410000000000",
      "begBal": "18255476410000000000",
      "endBal": "17950072890000000000",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "305403520000000000",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "",
      "spotPrice": 7616.14451,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "17950072890000000000",
      "endBalDiff": "",
      "totalIn": "",
      "totalOut": "305403520000000000",
      "totalOutLessGas": "305403520000000000",
      "amountNet": "-305403520000000000",
      "reconciled": true
    }

while there's no statement for YFI token associated with the transaction with index 305 in this block. The total sum of 305403520000000000 in the statement is wrong, though it's the sum of the values of the two transactions. Basically the first occurence of transaction for given token aggregates values of all the remaining transactions for that token and that address in the block.

Expected result: The ERC20 token transfers inside same block that involve the same address for which statements are generated should be correctly associated with appropriate transactions.

Jun 09 '22 21:06 artur-jablonski

Thanks for this excellent find. We knew about this. It's actually quite difficult to do because the chain itself quite literally does not report balances after each transaction, only after each block. In the ETH reconcilations, we do make partial reconcilations which are not technically correct because we have no ability query balances after the transaction only after the block (as you've discovered for tokens).

Two things:

We are porting all of our code from C++ to Go Lang. We've made the decision to not correct this particular problem until the code is fully ported. This will take a while because (a) we're careful, (b) it's hard, and (c) we have a few other things that are currently more important.

Erigon is writing a version they are calling Erigon 2. One of the amazing things they are going to add is per-transaction state (which includes balances) queries. So, it seems it would actually not only be much easier, but much more accurate to wait until Erigon has per-transaction queries.

I know this isn't what you want to hear, but I think this is how we're going to proceed.

Upshot: This won't get fixed until Erigon has per-transaction balance queries. Once they do, hopefully we're done porting and then it will be much easier, much more clear, and more accurate.

Jun 10 '22 01:06 tjayrush

I'll keep the issue open until it's completed.

Jun 10 '22 01:06 tjayrush

Thank you for explanation. I understand now that reconciliation only makes sense on block granularity at the moment.

Still, by looking at the code I think the core of the problem is that ERC20 reconciliation doesn't go after value transfers at all. It actually should be easier to perform partial reconciliation for ERC20 than for ETH since all transfers are in logs and you don't need to go for traces for that.

Jun 10 '22 08:06 artur-jablonski

You're right, but we're in the process of porting, so we decided to wait.

I totally invite you to make a PR though... :-). I'd be totally open to that.

We just have a pretty small team...

Jun 10 '22 10:06 tjayrush

Sure. I will have a jab a this.

Jun 10 '22 10:06 artur-jablonski

https://github.com/TrueBlocks/trueblocks-core/pull/2168

This is an attempt at fixing this. The particular case reported in this issue is now behaving correctly. I am not sure how to better test this. Have a look, tell me what you think.

Jun 10 '22 17:06 artur-jablonski

This is great. Thanks so much.

In answer to your question about testing. You can test things as follows:

In one terminal window, make && chifra serve
In another, make test-all

This will run about 1,500 tests (I'm not sure if they will all pass).

To add a new test, you can look in ~/src/dev-tools/testRunner/testCases/. For this case, you would add a line to the file ./apps/acctExport.csv. It should be fairly self-explanitory. You can copy and paste the line, change the test name and change the command.

It's not necessary for a PR, but it helps. I always add at least one test that tries to cover any new code paths, but if you add one, that helps too.

Cheers.

Jun 10 '22 22:06 tjayrush

Ok. I did my best to add the test case, but can't really verify it works locally cuz make test-all fails for me. @tjayrush can you have a look if what I added is any good? Cheerio

Jun 14 '22 13:06 artur-jablonski

Thank you so much for this.

I will run this test against your branch shortly. (Please disregard my previous post -- I ran the test against the wrong branch.)

Jun 14 '22 15:06 tjayrush

Sorry it's taking so long to merge your PR. I'm going to merge it, but I need to take the time to review it carefully. We have a pretty major bit of work going on right now, might take a week or two to complete. Once that's done, I'll try to merge your PR. I want to surround it with some more test cases as well.

Jun 14 '22 15:06 tjayrush

Sure, no probs. Wherever you have a moment to pick it up.

The output you posted is off by one record comparing to my output. So your second record is my first. I used first_record = 683 & max_records = 3 to limit the amount of data that is compared. So that would suggest that my index is missing one record... :thinking:

This is chifra list 0x6e4c6D9B0930073e958ABd2ABa516b885260b8Ff output:

https://gist.github.com/artur-jablonski/cd62277fa7125d60d9ceabdc8f8a83d8

Jun 14 '22 15:06 artur-jablonski

For completeness, my output:

{ "data": [
{
  "hash": "0x458ae31579274588f04850853ec71f67e1e21b2d4e72ada18fdc04352fc22295",
  "blockHash": "0xe83164313d56dd57fd718123c311cd65d58f22761899bfb1710c77e2d7f67878",
  "blockNumber": 14933657,
  "transactionIndex": 303,
  "timestamp": 1654793734,
  "from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
  "to": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
  "value": 0,
  "gas": 120000,
  "gasPrice": 104646629487,
  "maxFeePerGas": 185012912936,
  "maxPriorityFeePerGas": 1012000000,
  "input": "0xa9059cbb000000000000000000000000aca096d2e6492d7380c5d8e70d90e13eecc72dd4000000000000000000000000000000000000000000000000043998c6eeb4ec00",
  "isError": 0,
  "hasToken": 1,
  "receipt": {
    "contractAddress": "0x0",
    "gasUsed": 34696,
    "effectiveGasPrice": 104646629487,
    "logs": [
      {
        "address": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
        "logIndex": 494,
        "topics": [
          "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
          "0x0000000000000000000000006aed588ca2052ccfc907db8c24df4b7b95a29a5e",
          "0x000000000000000000000000aca096d2e6492d7380c5d8e70d90e13eecc72dd4"
        ],
        "data": "0x000000000000000000000000000000000000000000000000043998c6eeb4ec00",
        "articulatedLog": {
          "name": "Transfer",
          "inputs": {
            "_amount": "304442430000000000",
            "_from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
            "_to": "0xaca096d2e6492d7380c5d8e70d90e13eecc72dd4"
          }
        },
        "compressedLog": "{name:Transfer|inputs:{_amount:304442430000000000|_from:0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e|_to:0xaca096d2e6492d7380c5d8e70d90e13eecc72dd4}}"
      }
    ],
    "status": 1
  },
  "traces": [],
  "articulatedTx": {
    "name": "transfer",
    "inputs": {
      "amount": "304442430000000000",
      "recipient": "0xaca096d2e6492d7380c5d8e70d90e13eecc72dd4"
    },
    "outputs": {
      "val_0": ""
    }
  },
  "compressedTx": "{name:transfer|inputs:{amount:304442430000000000|recipient:0xaca096d2e6492d7380c5d8e70d90e13eecc72dd4}|outputs:{val_0:}}",
  "statements": [
    {
      "blockNumber": 14933657,
      "transactionIndex": 303,
      "timestamp": 1654793734,
      "assetAddr": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
      "assetSymbol": "WEI",
      "decimals": 18,
      "prevBlk": 14933657,
      "prevBlkBal": "568293581338250642905",
      "begBal": "568293581338250642905",
      "endBal": "568289950518793961953",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "3630819456680952",
      "reconciliationType": "partial-partial",
      "spotPrice": 1801.80999,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "568289950518793961953",
      "endBalDiff": "",
      "totalIn": "",
      "totalOut": "3630819456680952",
      "totalOutLessGas": "",
      "amountNet": "-3630819456680952",
      "reconciled": true
    },
    {
      "blockNumber": 14933657,
      "transactionIndex": 303,
      "timestamp": 1654793734,
      "assetAddr": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
      "assetSymbol": "YFI",
      "decimals": 18,
      "prevBlk": 14933656,
      "prevBlkBal": "17950072890000000000",
      "begBal": "17950072890000000000",
      "endBal": "17645630460000000000",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "304442430000000000",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "",
      "reconciliationType": "",
      "spotPrice": 7616.14451,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "17645630460000000000",
      "endBalDiff": "",
      "totalIn": "",
      "totalOut": "304442430000000000",
      "totalOutLessGas": "304442430000000000",
      "amountNet": "-304442430000000000",
      "reconciled": true
    }
  ],
  "gasCost": 3630819456680952,
  "etherGasCost": 0.003630819456680952,
  "function": "transfer",
  "gasUsed": 34696,
  "date": "2022-06-09 16:55:34 UTC",
  "ether": 0.000000000000000000,
  "encoding": "0xa9059cbb"
}
, {
  "hash": "0x9c07595dec64bb699ef24c1c5e5579edba940fb9596e38bd4465f5c840393aae",
  "blockHash": "0xe83164313d56dd57fd718123c311cd65d58f22761899bfb1710c77e2d7f67878",
  "blockNumber": 14933657,
  "transactionIndex": 304,
  "timestamp": 1654793734,
  "from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
  "to": "0x3506424f91fd33084466f402d5d97f05f8e3b4af",
  "value": 0,
  "gas": 65226,
  "gasPrice": 104634629487,
  "maxFeePerGas": 197965383276,
  "maxPriorityFeePerGas": 1000000000,
  "input": "0xa9059cbb000000000000000000000000cc34cc875d2d052f5b1c19652b5e71d6f90c7f7d000000000000000000000000000000000000000000000010f34de690046d8400",
  "isError": 0,
  "hasToken": 1,
  "receipt": {
    "contractAddress": "0x0",
    "gasUsed": 54355,
    "effectiveGasPrice": 104634629487,
    "logs": [
      {
        "address": "0x3506424f91fd33084466f402d5d97f05f8e3b4af",
        "logIndex": 495,
        "topics": [
          "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
          "0x0000000000000000000000006aed588ca2052ccfc907db8c24df4b7b95a29a5e",
          "0x000000000000000000000000cc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
        ],
        "data": "0x000000000000000000000000000000000000000000000010f34de690046d8400",
        "articulatedLog": {
          "name": "Transfer",
          "inputs": {
            "_amount": "312679827610000000000",
            "_from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
            "_to": "0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
          }
        },
        "compressedLog": "{name:Transfer|inputs:{_amount:312679827610000000000|_from:0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e|_to:0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d}}"
      }
    ],
    "status": 1
  },
  "traces": [],
  "articulatedTx": {
    "name": "transfer",
    "inputs": {
      "amount": "312679827610000000000",
      "recipient": "0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
    },
    "outputs": {
      "val_0": ""
    }
  },
  "compressedTx": "{name:transfer|inputs:{amount:312679827610000000000|recipient:0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d}|outputs:{val_0:}}",
  "statements": [
    {
      "blockNumber": 14933657,
      "transactionIndex": 304,
      "timestamp": 1654793734,
      "assetAddr": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
      "assetSymbol": "WEI",
      "decimals": 18,
      "prevBlk": 14933657,
      "prevBlkBal": "568289950518793961953",
      "begBal": "568289950518793961953",
      "endBal": "568271271877180349122",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "5687415285765885",
      "reconciliationType": "partial-nextdiff",
      "spotPrice": 1801.80999,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "568284263103508196068",
      "endBalDiff": "12991226327846946",
      "totalIn": "",
      "totalOut": "5687415285765885",
      "totalOutLessGas": "",
      "amountNet": "-5687415285765885",
      "reconciled": false
    },
    {
      "blockNumber": 14933657,
      "transactionIndex": 304,
      "timestamp": 1654793734,
      "assetAddr": "0x3506424f91fd33084466f402d5d97f05f8e3b4af",
      "assetSymbol": "0x35",
      "decimals": 18,
      "prevBlk": 14933656,
      "prevBlkBal": "791184699027030000000000",
      "begBal": "791184699027030000000000",
      "endBal": "790872019199420000000000",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "312679827610000000000",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "",
      "reconciliationType": "",
      "spotPrice": 0.11947,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "790872019199420000000000",
      "endBalDiff": "",
      "totalIn": "",
      "totalOut": "312679827610000000000",
      "totalOutLessGas": "312679827610000000000",
      "amountNet": "-312679827610000000000",
      "reconciled": true
    }
  ],
  "gasCost": 5687415285765885,
  "etherGasCost": 0.005687415285765885,
  "function": "transfer",
  "gasUsed": 54355,
  "date": "2022-06-09 16:55:34 UTC",
  "ether": 0.000000000000000000,
  "encoding": "0xa9059cbb"
}
, {
  "hash": "0x8c1703f5dee17242784c912f208f55508cf003f2b3b0ca9070d9556a619df186",
  "blockHash": "0xe83164313d56dd57fd718123c311cd65d58f22761899bfb1710c77e2d7f67878",
  "blockNumber": 14933657,
  "transactionIndex": 305,
  "timestamp": 1654793734,
  "from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
  "to": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
  "value": 0,
  "gas": 120000,
  "gasPrice": 104634629487,
  "maxFeePerGas": 203165801707,
  "maxPriorityFeePerGas": 1000000000,
  "input": "0xa9059cbb000000000000000000000000cc34cc875d2d052f5b1c19652b5e71d6f90c7f7d00000000000000000000000000000000000000000000000000036a1b340c1400",
  "isError": 0,
  "hasToken": 1,
  "receipt": {
    "contractAddress": "0x0",
    "gasUsed": 51784,
    "effectiveGasPrice": 104634629487,
    "logs": [
      {
        "address": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
        "logIndex": 496,
        "topics": [
          "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef",
          "0x0000000000000000000000006aed588ca2052ccfc907db8c24df4b7b95a29a5e",
          "0x000000000000000000000000cc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
        ],
        "data": "0x00000000000000000000000000000000000000000000000000036a1b340c1400",
        "articulatedLog": {
          "name": "Transfer",
          "inputs": {
            "_amount": "961090000000000",
            "_from": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
            "_to": "0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
          }
        },
        "compressedLog": "{name:Transfer|inputs:{_amount:961090000000000|_from:0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e|_to:0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d}}"
      }
    ],
    "status": 1
  },
  "traces": [],
  "articulatedTx": {
    "name": "transfer",
    "inputs": {
      "amount": "961090000000000",
      "recipient": "0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d"
    },
    "outputs": {
      "val_0": ""
    }
  },
  "compressedTx": "{name:transfer|inputs:{amount:961090000000000|recipient:0xcc34cc875d2d052f5b1c19652b5e71d6f90c7f7d}|outputs:{val_0:}}",
  "statements": [
    {
      "blockNumber": 14933657,
      "transactionIndex": 305,
      "timestamp": 1654793734,
      "assetAddr": "0x6aed588ca2052ccfc907db8c24df4b7b95a29a5e",
      "assetSymbol": "WEI",
      "decimals": 18,
      "prevBlk": 14933657,
      "prevBlkBal": "568302725253886882348",
      "begBal": "568302725253886882348",
      "endBal": "568271271877180349122",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "5418399653354808",
      "reconciliationType": "partial-nextdiff",
      "spotPrice": 1801.80999,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "568297306854233527540",
      "endBalDiff": "26034977053178418",
      "totalIn": "",
      "totalOut": "5418399653354808",
      "totalOutLessGas": "",
      "amountNet": "-5418399653354808",
      "reconciled": false
    },
    {
      "blockNumber": 14933657,
      "transactionIndex": 305,
      "timestamp": 1654793734,
      "assetAddr": "0x0bc529c00c6401aef6d220be8c6ea1667f6ad93e",
      "assetSymbol": "YFI",
      "decimals": 18,
      "prevBlk": 14933657,
      "prevBlkBal": "17645630460000000000",
      "begBal": "17645630460000000000",
      "endBal": "17644669370000000000",
      "amountIn": "",
      "internalIn": "",
      "selfDestructIn": "",
      "minerBaseRewardIn": "",
      "minerNephewRewardIn": "",
      "minerTxFeeIn": "",
      "minerUncleRewardIn": "",
      "prefundIn": "",
      "amountOut": "961090000000000",
      "internalOut": "",
      "selfDestructOut": "",
      "gasCostOut": "",
      "reconciliationType": "",
      "spotPrice": 7616.14451,
      "priceSource": "uniswap",
      "begBalDiff": "",
      "endBalCalc": "17644669370000000000",
      "endBalDiff": "",
      "totalIn": "",
      "totalOut": "961090000000000",
      "totalOutLessGas": "961090000000000",
      "amountNet": "-961090000000000",
      "reconciled": true
    }
  ],
  "gasCost": 5418399653354808,
  "etherGasCost": 0.005418399653354808,
  "function": "transfer",
  "gasUsed": 51784,
  "date": "2022-06-09 16:55:34 UTC",
  "ether": 0.000000000000000000,
  "encoding": "0xa9059cbb"
}
] }

Jun 14 '22 15:06 artur-jablonski

Thank you for explanation. I understand now that reconciliation only makes sense on block granularity at the moment.

Still, by looking at the code I think the core of the problem is that ERC20 reconciliation doesn't go after value transfers at all. It actually should be easier to perform partial reconciliation for ERC20 than for ETH since all transfers are in logs and you don't need to go for traces for that.

We're currently re-writing the accounting code, and we're now taking into account the fact that all token transfers get an Event (or at least all of them should -- not convinced that all of them do). Re-write is coming soon.

Nov 04 '22 15:11 tjayrush

@artur-jablonski This work has been completed and is now in develop. We will be pushing it to master later today or tomorrow.

Nov 21 '22 03:11 tjayrush

trueblocks-core trueblocks-core copied to clipboard

chifra export: Intrablock transactions for same erc20 tokens and address are collapsed into one transaction.

trueblocks-core
trueblocks-core copied to clipboard