The real-world availability of MESSAGE_LIMIT and OP_RETURN, and API quirks

[lannocc]

After lots of testing, head-scratching, and aggravation, I find the following to be true:

1. 75 bytes is the actual maximum message size.
2. Multiple OP_RETURNS in a single transaction are not supported.

I have not tested whether the above conditions hold true beyond the 2 web APIs used by BitCash or if they also hold when sending directly to a node. Also, these APIs react differently when the above conditions are not met.

The Bitcoin.com API will simply reject with HTTP status 400 when a transaction with OP_RETURN message size > 75 bytes is received, or if there is more than one OP_RETURN in the transaction vouts. The Blockdozer.com API will accept these, however they do not get relayed anywhere else as far as I can tell and simply remain unconfirmed for a long time (sometimes dropping off but other times reappearing).

Additional note: the Blockdozer API is a bit unreliable. It once returned status 504 (Gateway Timeout) even though a new unconfirmed transaction actually did appear on their site, though in BitCash this raises an exception, so no transaction id was returned. Unconfirmed transactions are showing up twice on Blockdozer. Also, I currently have a list of unconfirmed transactions on the Blockdozer site that are all showing a clearly bogus fee of 184467440737.0945 BCH!

[AustEcon]

Have you tried it with the different OP_PUSH code for when you go above 75 bytes? (As per bitcoin script wiki)? I think 75 bytes is the limit before you need to specify an OP_PUSH (which then gets up up to like 256 bytes). I haven't added this to my bchpusher module yet because I'm lazy and have other stuff atm to do...

Agreed about the multiple ORs in one txn... doesn't relay. At least not via current bitcash APIs. Don't know if p2p nodes would accept or not but definitely "non-standard"

[lannocc]

I admit I'm still pretty ignorant to the actual standard(s) and protocol requirements, so I have not tried it with OP_PUSH. That's a relief if we can actually get more than 75 bytes. I'll have to dig into the BitCash code a little deeper, and the wiki you referenced, to see what I can do about this and try out your suggestion.

[AustEcon]

Oh sweet. It's an easy fix!! I can assure you!! Will be like 4 - 6 lines of code max in the ? construct_output_block().

Just need to add in special case > 75 bytes and they even have provision to go > 256 bytes in that OP codes.

I'm out and about (not at my computer) atm so can't give you much more than this. But my feeling is that all will be okay :)

Edit: It's called OP_PUSHDATA_1 (0x4c). The byte that follows specifies the number of bytes of PUSHDATA to follow. You need this for >75 bytes.

[lannocc]

Thanks, I'm on it!

Do you have any insight about the rejection of transactions with multiple OP_RETURN vouts? The Script wiki says:

Currently it is usually considered non-standard (though valid) for a transaction to have more than one OP_RETURN output or an OP_RETURN output with more than one pushdata op.

However the Bitcoin.com API always rejects this with my tests, even when both OP_RETURN payloads are very small.

[AustEcon]

All I can say on the multiple ORs front, is that I too have had a sporadic go at testing this out and have not had any success. I do wonder if a p2p node might accept though... (like if it came from electrum wallet or something). I'm pretty sure miners themselves have a kind of special priviledge where they can do multiple ORs in one txn and it will be accepted by others if they find the golden hash and relay the block. I don't know though what the wider situation is though around miners accepting it in the usual sense p2p.

[AustEcon]

I've got it working. see 8a774d76a65983809d6453a2d52f2046bf126b4394325225fa27b57f6d49b42d for evidence. of 'hello'*40 successfully sent.

I'll do PR now so you can have a look

[lannocc]

I just did a successful test too. Waiting for it to confirm to be sure.

[lannocc]

    # Blockchain storage
    else:
        size = len(dest)
        script = OP_RETURN

        if size > MESSAGE_LIMIT_SANS_PUSH:
            script += OP_PUSHDATA_1

        script += size.to_bytes(1, byteorder='little') + dest

        output_block += b'\x00\x00\x00\x00\x00\x00\x00\x00'

[AustEcon]

Yeah mine is: if custom_pushdata is False: script = OP_RETURN

            if len(dest) <=75: # (https://en.bitcoin.it/wiki/Script)
                script += len(dest).to_bytes(1, byteorder='little') +  dest
            elif len(dest)<=256:
                script += b"\x4c" + len(dest).to_bytes(1, byteorder='little') +  dest # OP_PUSHDATA1 format
            else:
                script += b"\x4d" + len(dest).to_bytes(2, byteorder='little') +  dest # OP_PUSHDATA2 format

            output_block += b'\x00\x00\x00\x00\x00\x00\x00\x00'

[lannocc]

I think yours might be more future-proof, moving into OP_PUSHDATA_2 :)

This got me curious though, since the size has been appended immediately after OP_RETURN when there is no OP_PUSHDATA, how does a node parse the byte immediately following OP_RETURN to know whether it's another OP vs the payload size? Now I understand the 75 byte message limit. A message size of 76 bytes would look exactly like OP_PUSHDATA_1 which is opcode 76.

[lannocc]

Have you seen https://github.com/grantcoin/grantcoin/blob/master/contrib/counterpartyd/lib/bitcoin.py ? That serialize() function is a beauty. Notice how it too seems to imply that multiple OP_RETURN is supported. I'm going to try this against my running node.

[AustEcon]

I hadn't no. But yes that IS a beauty. haha. They have that nice op_push() in there too which does what we have just been talking about (they go up to OP_PUSHDATA4... do we want it?? Can't have them outdoing us...)

And yeah... good question about the parsing over the data... I don't know.

[lannocc]

I answered my question just as I was asking it. It's a hack, really, since pushdata ops are seemingly the only op type allowed to follow an op_return... they simply allow no op for payload size only up to 1 less than the opcode of the first pushdata, so it's unambiguous.

[lannocc]

You know you want to add that PUSHDATA_4 now... otherwise you're only going halfway on the not-expected-to-work-now-but-maybe-later ops :)

[AustEcon]

ahh right. So do you think that the choice of 75 was simply decided by how much room they had left for op codes up to a limit of 255... + a bit of extra room in case they want to add in more later on... Otherwise 75 just seems very arbitrary and a strange choice for where to settle on... I guess if you have enough leftover space for op codes up the other end of the scale... then you might as well make all of those messages <75 save an extra byte by making no_op default for as wide of a spectrum as you can allow?? (only way I can rationalize it... all for the sake of saving one byte of data... just because they can)

[lannocc]

Yeah it must be something like that. I only just realized that there are no opcodes 1-75, so it is in fact defined that anytime you expect to read an opcode and see a non-zero value < 76 then that value is meant to be the length for data to push onto the stack. Or, put differently, opcodes 1-75 are all PUSHDATA ops that don't require a size, as their opcode number IS the size.

I'm with you, it must have been a consideration to save space instead of reserving those numbers for future opcodes.

[ghost]

Wow, this is awesome. Sorry for the lack of response, I've been traveling. Feel free to send an email if I'm not checking in, say every 24 hours.

[lannocc]

Only issue remaining here is the multiple OP_RETURN support. I recommend we simply raise an exception for now if multiple OP_RETURN outputs would be created, since the API does not accept them. I can submit a pull request that does this if you agree.

[AustEcon]

I think that's fair enough. Saves ppl trying to relay txn and wondering why it's not working.

[devalbo]

I love the activity going on with bitcash. I've been using it for a project of mine and been able to successfully create transactions on testnet with multiple OP_RETURNS (see https://test-bch-insight.bitpay.com/api/tx/af6cbfdf7160d59f50199f81fa0f3c9de8710104b2cdf61ffdcd2ec0729118ae ). If you make multiple OP_RETURNS raise an exception, could you make suppressing the exception an option since it is still accepted by some miners (though it is non-standard)?

[merc1er]

Closing this (very) old issue as nearly everything mentioned has changed:

• rest.Bitcoin.com is no longer operational
• Blockdozer is no longer operational
• Bitcoin Cash now supports multiple OP_RETURNS: https://bch.info/en/chips (search for "multiple op_returns"), however BitCash does not currently support including more than one
• The size of the data in OP_RETURN was increased (I don't recall exactly the number, but it was 220 bytes at some point, then increased again I believe)

Feel free to open new issues if needed 👍🏻