Different ways of linking produce different bytecode

[xuhcc]

Description

The bytecode produced by solc when linking of libraries is done during the compilation is not the same as bytecode produced when linking is done after the compilation.

Environment

• Compiler version: 0.5.17+commit.d19bba13.Linux.g++
• Target EVM version (as per compiler settings): default
• Framework/IDE (e.g. Truffle or Remix): N/A
• EVM execution environment / backend / blockchain client: N/A
• Operating system: Arch Linux

Steps to Reproduce

Standard JSON input:

{
  "language": "Solidity",
  "sources": {
    "Contract.sol": {
      "content": "pragma solidity ^0.5.0;\n\nimport {Library} from \"./Library.sol\";\ncontract Contract {\n    function test(uint256[] memory array) public pure returns (uint256) {\n        return Library.test(array);\n    }}\n"
    },
    "Library.sol": {
      "content": "pragma solidity ^0.5.0;\n\n\nlibrary Library {\n    function test(uint256[] memory input) public pure returns (uint256);\n}\n\n"
    }
  },
  "settings": {
    "metadata": {
      "useLiteralContent": true
    },
    "optimizer": {
      "enabled": true,
      "runs": 20
    },
    "libraries": {
      "Contract.sol": {
        "Library": "0xd0cd59b43bec6d78043a366165f6352824a67586"
      }
    },
    "outputSelection": {
      "*": {
        "*": [
          "abi",
          "evm.bytecode",
          "evm.deployedBytecode",
          "evm.methodIdentifiers"
        ],
        "": [
          "id",
          "ast"
        ]
      }
    }
  }
}

Case 1. Read input file and compile:

$ cat solc-input.json | ./solc-static-linux --standard-json | jq -r '.contracts["Contract.sol"].Contract.evm.deployedBytecode.object'
608060405234801561001057600080fd5b506004361061002b5760003560e01c8063ca16068414610030575b600080fd5b6100d16004803603602081101561004657600080fd5b810190602081018135600160201b81111561006057600080fd5b82018360208201111561007257600080fd5b803590602001918460208302840111600160201b8311171561009357600080fd5b9190808060200260200160405190810160405280939291908181526020018383602002808284376000920191909152509295506100e3945050505050565b60408051918252519081900360200190f35b600073d0cd59b43bec6d78043a366165f6352824a6758663ca160684836040518263ffffffff1660e01b81526004018080602001828103825283818151815260200191508051906020019060200280838360005b8381101561014f578181015183820152602001610137565b505050509050019250505060206040518083038186803b15801561017257600080fd5b505af4158015610186573d6000803e3d6000fd5b505050506040513d602081101561019c57600080fd5b50519291505056fea265627a7a7231582024de7d6243db640302dde781dc0d79f209d4e34739009264fb27c0498a5702fa64736f6c63430005110032

Case 2. Read input file, remove libraries from it, compile and link after the compilation:

$ cat solc-input.json | jq -r 'del(.settings.libraries)' | ./solc-static-linux --standard-json | jq -r '.contracts["Contract.sol"].Contract.evm.deployedBytecode.object' | ./solc-static-linux - --link --libraries "Library.sol:Library:0xd0cd59b43bec6d78043a366165f6352824a67586"
608060405234801561001057600080fd5b506004361061002b5760003560e01c8063ca16068414610030575b600080fd5b6100d16004803603602081101561004657600080fd5b810190602081018135600160201b81111561006057600080fd5b82018360208201111561007257600080fd5b803590602001918460208302840111600160201b8311171561009357600080fd5b9190808060200260200160405190810160405280939291908181526020018383602002808284376000920191909152509295506100e3945050505050565b60408051918252519081900360200190f35b600073d0cd59b43bec6d78043a366165f6352824a6758663ca160684836040518263ffffffff1660e01b81526004018080602001828103825283818151815260200191508051906020019060200280838360005b8381101561014f578181015183820152602001610137565b505050509050019250505060206040518083038186803b15801561017257600080fd5b505af4158015610186573d6000803e3d6000fd5b505050506040513d602081101561019c57600080fd5b50519291505056fea265627a7a723158208cebe246c29cf925d04807debe830fc2b44a2d5c550b2b21ba74bc112c1997b664736f6c63430005110032
Linking completed.

The results are different.

[chriseth]

This is true because the supplied library addresses are considered compiler input. After all, the compiler actually inserts the addresses into its output.

This is related to the metadata hash being part of the bytecode (I assume only the metadata hash is different) to be able to verify compilations.

[xuhcc]

@chriseth The part which differs is 64-character string near the end.

The problem is that if contract was compiled, linked and deployed (case 2), it's not possible to verify it using Standard JSON method on Etherscan.

[chriseth]

This looks to me like a limitation of etherscan - it should allow both ways to link the binary.

[xuhcc]

I agree, but it's just an example. I think many developers have a justified expectation that given the same source code one should get the same bytecode no matter how the libraries are linked. The fact that different ways of linking produce different bytecode is surprising and can cause issues downstream that are hard to debug (like in etherscan verification case).

[cameel]

I don't know if we'll be changing this behavior but now it's at least documented in the section about linker (#10291).

[cameel]

We talked about this on the call today:

• We should have a documentation section going more in depth about issues related to source code verification and bytecode reproducibility.
• We're probably going to get rid of the --link option eventually because it's not used much in practice (the high cost of delegatecall discourages people from using external libraries) but this problem still potentially affects any external tools that do linking on their own.
• Removing the --libraries option from the metadata would make the bytecode linked using both methods identical but that information is useful and we don't think removing it is a good idea.
• @axic proposed that we should be more explicit in the metadata about the existence of unlinked references.
• We could have the --link option rewrite the metadata and in the end produce the same bytecode you would get if it you supplied --libraries during compilation. We don't really want to do that though because it would be hard to use in practice, making the --link option even less useful - you'd have to supply the original metadata as input for it to work.

We haven't come to any firm decision yet but I think that the current behavior is more likely to just be more extensively documented and slightly tweaked than changed in a radical way.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 90 days. It will be automatically closed in 7 days.

[github-actions[bot]]

Hi everyone! This issue has been automatically closed due to inactivity. If you think this issue is still relevant in the latest Solidity version and you have something to contribute, feel free to reopen. However, unless the issue is a concrete proposal that can be implemented, we recommend starting a language discussion on the forum instead.

[cameel]

Assuming we won't drop standalone linking functionality somewhere along the way, this issue would be solved by what I described in https://github.com/ethereum/solidity/issues/14986#issuecomment-2037698693. I.e. linking could be a distinct stage at the end and --libraries would always be a part of the metadata of that stage, no matter whether supplied initially or only when linking is performed.