mandatory-script-verify-flag-failed (Push value size limit exceeded) error occurs when sending raw tx with Taproot inscriptions exceeding ~500KB

[ongrid]

Update: The root cause of the mandatory-script-verify-flag-failed (Push value size limit exceeded) error when processing the sendrawtransaction command by Bitcoin Core, is the script tokensize limit defined in bitcoind MAX_SCRIPT_ELEMENT_SIZE = 520 constant https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27

If you encounter this error, you'll need to break down long data into smaller tokens (I used 512 bytes for better kilobyte-alignment), and then use these minimized tokens in the script. Here’s a function and an example of how to use it, shown below.

# To avoid `mandatory-script-verify-flag-failed (Push value size limit exceeded)` in bitcoin-core
# See https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27
# and https://github.com/karask/python-bitcoin-utils/issues/69
def chunk_script_element(data: bytes):

    BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE = 512

    for i in range(0, len(data), BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE):
        yield data[i:i+BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE]

Example use for ordinals case

            taproot_script_p2pk = Script(
                [
                    priv_key.get_public_key().to_x_only_hex(),
                    "OP_CHECKSIG",
                    "OP_0",
                    "OP_IF",
                    "ord".encode("utf-8").hex(),
                    "01",
                    MIME_TYPE.encode("utf-8").hex(),
                    "OP_0",
                    *[chunk.hex() for chunk in chunk_script_element(content)],
                    "OP_ENDIF",
                ]
            )

My suspicions that the problem was with the functions in the Script class were not confirmed.

Script.to_bytes() function and especially _op_push_data seems to populate OP_PUSHDATA_2 length padding using incorrect endianness when handling scripts containing hexadecimal data exceeding 76 bytes.

This may lead to various issues with script execution like mandatory-script-verify-flag-failed (Opcode missing or not understood) on broadcasting raw transaction. The to_hex() and from_raw() serialization functions are exhibiting unreliable behavior when handling scripts containing hexadecimal data exceeding 76 bytes. This issue disrupts the consistency of data posted by PUSHDATA2, leading to padding errors.

I noticed the incorrect length when I investigated errors of "Push value size limit exceeded":

mandatory-script-verify-flag-failed (Push value size limit exceeded)
[02000000000101c076929da8ada601dd2d16ce6ce269df47b86c23914fb56cfdc9eaede9c93e6c0000000000ffffffff025e01000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d32cf030000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d03404af188ce6dfda88a894e1cd63f4949f5957095e9eb552932a38e67eb51e5f22aadf4352d4c05f95dc7a5bdd2387f38ba432e0a7b1bfdcac888f239173fdf9b9dfd530220c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000]

When a script is created with hex data longer than 76 bytes, the PUSHDATA2 length bytes are not handled correctly:

Reproduction

privkey_x_only_hex = "de" * 31
content = "deadbeef" * 100
script = Script(
    [
        privkey_x_only_hex,
        "OP_CHECKSIG",
        "OP_0",
        "OP_IF",
        content,
        "OP_ENDIF",
    ]
)
hex_dump = script.to_hex()

hex_dump value:

1fdededededededededededededededededededededededededededededededeac00634d9001deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef68

Actual content's datalength is 400B

>len(bytes.fromhex(content))
400
>hex(len(bytes.fromhex(content)))
'0x190'

('0x190' in hex)

_op_push_data generates the following sequence to wrap hex content setting length to hex(9001) that has decimal value of 36865.

4d    9001     deadbeefdeadbeef....
--    -----    ------------------
|      |          Data payload
|      | Length
|
OP_PUSHDATA2

PUSHDATA2 spec illustrates datalength should be formatted in normal (big-endian) order:

Examples: 0x4D 0x0100 <256 byte data item> - would leave the 256 byte data item on the stack.

So in our case, it should be

4d    0190     deadbeefdeadbeef....

Such misalignment (when length > max push size) seems to lead to the following trace on execution (example from live tx)

mandatory-script-verify-flag-failed (Push value size limit exceeded)
[02000000000101c076929da8ada601dd2d16ce6ce269df47b86c23914fb56cfdc9eaede9c93e6c0000000000ffffffff025e01000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d32cf030000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d03404af188ce6dfda88a894e1cd63f4949f5957095e9eb552932a38e67eb51e5f22aadf4352d4c05f95dc7a5bdd2387f38ba432e0a7b1bfdcac888f239173fdf9b9dfd530220c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000]

When length is less than max_push_size, the cursor points to a byte of data that is treated as an opcode, resulting in a variety of OP-specific errors.

Preliminary root cause

My research show that root cause may be in the _op_push_data parser here, especially in pack formatter and direction of endianness:

https://github.com/karask/python-bitcoin-utils/blob/b10f493be4b3e2eadd9d91bf74e9cd04120a9f41/bitcoinutils/script.py#L302C1-L307C1

and fix can be as obvious as

 -           return b"\x4d" + struct.pack("<H", len(data_bytes)) + data_bytes
 +          return b"\x4d" + struct.pack(">H", len(data_bytes)) + data_bytes
            

But I'm unable to verify this solution due to another issue #68