BIP40: Stratum wire protocol

[ben221199]

The Stratum wire protocol has a long history since @slush0 introduced it. However, the protocol never got standardized in a formal way, so many implementations have been based on incomplete documents or on other implementations. With this document I finally want to give Stratum its place between the other BIPs, so that developers can just read this document and don't have to search through years of code or dead pages that need to be revived with Wayback Machine.

[ben221199]

Yesterday, I sent two registration requests to IANA to get some information officially egistered. The registries are:

• Media Types (https://www.iana.org/assignments/media-types/media-types.xhtml)
• WebSocket Subprotocol Name Registry (https://www.iana.org/assignments/websocket/websocket.xml#subprotocol-name)

Today, I got a response that for both registrations some things have to be done first, before IANA can add these records to their registry. In case of the Media Types, this is process is a little bit more difficult. For the WebSocket Subprotocol Name Registry it is just letting them know when this pull request is merged and I have a permalink to the specification.

Also, it is possible that I will send more registration requests for other IANA registries too. For example, the Service Name and Transport Protocol Port Number Registry (https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml) is a possible candidate. In that case, I will post new comments about that too.

[jonatack]

This BIP refers to BIP41, The Stratum mining protocol, for which a BIP number was assigned and an entry exists in the README, but there doesn't seem to be pull request to add the BIP draft -- any update on that?

[ben221199]

Hi @jonatack, thank you for feedback. I will take a look at it. I didn't have time to work on BIP 40 lately. Yes, both BIP 40 and BIP 41 are already assigned by the README and BIP 41 is also mentioned in this BIP 40. The goal is to finish BIP 40 (Stratum wire protocol) first and then start working on BIP 41 (Stratum mining protocol), referring back to BIP 40, because the mining protocol is based on the wire protocol.

[ben221199]

The media type application/stratum has been registered by IANA at Media Types (https://www.iana.org/assignments/media-types/media-types.xhtml).

[jonatack]

@ben221199 This draft might be missing the following:

- [ ] Backwards compatibility section describing incompatibilities, severity, and suggest mitigations
- [ ] Reference implementation (necessary before status "Final", but not necessary to be accepted)

[ben221199]

Where should I place that?

[jonatack]

Where should I place that?

They are generally the last two sections in other BIPs.

[plebhash]

It seems BIP41 doesn't have any PR yet, so I'll write a comment on this one.

My goal is to inform BIP41 authors (@ben221199 or whoever else volunteers) that the Stratum Reference Implementation community is aiming for a similar effort for Stratum V2

[ben221199]

Hi @plebhash, thanks for commenting. You are correct that there is no BIP 41 pull request yet. This is because BIP 41 is an implementation of BIP 40 specifically for mining. I will start working on that one when BIP 40 is finished.

As far as I'm aware, Stratum V2 is specifically for mining. When people want to make improvements on Stratum V1, I think that is solely a BIP 41 improvement, because BIP 40 only describes the wire protocol (the basics) and the Electrum implementation, not the mining one.

I think there are some things that could be done:

• Create a BIP for Stratum V2.
• Make a decision if you want to create a new BIP for the V1 improvements, or if you want those improvements merged into BIP 41.

Also, can you give me some examples on what those improvments could be?

Thanks

[plebhash]

Create a BIP for Stratum V2.

that is an interesting idea!

[plebhash]

@ben221199 on the context of BIP40, could you provide some wayback machine link for the original (non-mining) stratum wire protocol?

this is a very interesting unknown fact in the mining community, and everyone I tell about this is surprised about this historical fact

[Davidson-Souza]

Hi @ben221199. Thank you for taking time to document the stratum protocol. However, I have some comments about your description of the electrum flavor of stratum. I'm not sure some of the specified endpoints are actually part of their protocol, for example: blockchain.address is long deprecated in favor of blockchain.scripthash. They also don't have any method to get a full block.

Here is a doc of the latest protocol and all methods they currently support.

[ben221199]

Hi @plebhash, the second message on https://bitcointalk.org/?topic=557866 is one of the source I have used. There are some other sources too, but I think this is the most important one.

[ben221199]

Hi @Davidson-Souza, Stratum is invented by @slush0 especially for Electrum. Stratum wouldn't be here without Electrum and Electrum would possibly not have been a success without Stratum. I think it is okay to have Electrum commands specified next to the more general commands like discovery.

I'm aware of the fact that blockchain.address has been deprecated by blockchain.scripthash. However, the Electrum commands specify also a versioning system (where Stratum itself does not), so I think it is also good to have every version documented. Technically, it is possible to write a client and/or server that implements the Stratum protocol with Electrum commands and support multiple protocol versions. Removing deprecated commands seems to be a bad practice in my eyes.

[Davidson-Souza]

Hi @Davidson-Souza, Stratum is invented by @slush0 especially for Electrum. Stratum wouldn't be here without Electrum and Electrum would possibly not have been a success without Stratum. I think it is okay to have Electrum commands specified next to the more general commands like discovery.

I'm aware of the fact that blockchain.address has been deprecated by blockchain.scripthash. However, the Electrum commands specify also a versioning system (where Stratum itself does not), so I think it is also good to have every version documented. Technically, it is possible to write a client and/or server that implements the Stratum protocol with Electrum commands and support multiple protocol versions. Removing deprecated commands seems to be a bad practice in my eyes.

Hi, thank you for your reply. I do agree with the general direction you propose, I was a bit confused that this is not clear in the text. My understanding after reading it is that it defines the currently used commands by wallets and servers, but this is not the case.

[ben221199]

@Davidson-Souza If you see improvements in text, don't hesitate to give suggestions. GitHub makes it possible to make comments on code lines. 😄

[ben221199]

Okay, I have to pick up this task again.

[ben221199]

Some information about old commits...

Oldest commits with same server code:

• https://github.com/spesmilo/electrum/commit/f0aae3cc0f587994c86c5bcbc083829c524d3b62 (Client)
• https://github.com/spesmilo/electrum-server/commit/4572ac0fa026b07665998d50991b6b61be6a07e5 (Server)

Server code removed from client repository:

• https://github.com/spesmilo/electrum/commit/2e5d015917be5989f82d94b9c18c68b40f443da3 (Client)

First appearance of Stratum in Electrum source code:

• https://github.com/spesmilo/electrum/commit/0fc0bdd667559da3b6bb6b58018de56ecbdfa94f (Client)
• https://github.com/spesmilo/electrum-server/commit/21b133f4706a55f317e322a6b2e593b5a0750707 (Server)

All commits where protocol versions changed:

• https://github.com/spesmilo/electrum-server/commit/f5e9e2ef81050ba5d0e0798f86be2be84054a862 (Server; 0.1)
• https://github.com/spesmilo/electrum-server/commit/cf7aed705dbfeaa8b9e56a0649fd70f48d80d33f (Server; 0.2)
• https://github.com/spesmilo/electrum-server/commit/6a80f4b4a5315ed9d00f2b3117a5e0d447465cda (Server; 0.3)
• https://github.com/spesmilo/electrum-server/commit/d600c3be009f131a35a622bc83e1ca9d40c241d7 (Server; 0.4)
• https://github.com/spesmilo/electrum/commit/8ed812830f8880d02ffb521ff8835bff71d2fa24 (Client; 0.5)
• https://github.com/spesmilo/electrum-server/commit/c293b5a5d4e71f555981ba934ec53dedcce36fea (Server; 0.5)
• https://github.com/spesmilo/electrum/commit/8ec6a60daf7b0f3019677f27a20f59d311730c0f (Client; 0.6)
• https://github.com/spesmilo/electrum-server/commit/1abbd5675d40b85eb339c90c4082cdc22e620a86 (Server; 0.6)
• https://github.com/spesmilo/electrum-server/commit/24aa687775d6eac180088f72b2e8226cb572d7ef (Server; 0.7)
• https://github.com/spesmilo/electrum-server/commit/a71f02d33aceec0b70bb365c7d89663a55c47f0a (Server; 0.8)
• https://github.com/spesmilo/electrum-server/commit/fdf26dcd063af1cc68adcaf278c7d0180296d6c5 (Server; 0.9)
• https://github.com/spesmilo/electrum/commit/118052d81597eff3eb636d242eacdd0437dabdd6 (Client; 0.10)
• https://github.com/spesmilo/electrum-server/commit/1774ec1496ac8545b4d1b6b88f53d20d966afa29 (Server; 0.10)
• https://github.com/spesmilo/electrum/commit/68873d92f9973b3d6befe9d2b69f67be71217501 (Client; 0.11)
• https://github.com/spesmilo/electrum-server/commit/2baa164fc842bf39309c78c15bf89ef2f90a7b95 (Server; 1.0)
• https://github.com/spesmilo/electrum/commit/7f9b473a988fc8461750f4f8f1622c429d11e0df (Client; 1.1)
• https://github.com/spesmilo/electrum/commit/c3f3843cc3f31862f7e3bea1126cb42150c405c4 (Client; 1.2)
• https://github.com/spesmilo/electrum/commit/8f36c9167de1bfd2505ef5516cd43661962b9c27 (Client; 1.4)

[ben221199]

And for the ElectrumX server...

Protocol Minimum:

• https://github.com/spesmilo/electrumx/commit/151da40d5b6be87bb95b7c8641262138f0333163 (ElectrumX; 1.0)
• https://github.com/spesmilo/electrumx/commit/135ab68f7419b36d4d565eb47c254ac3240831bd (ElectrumX; 0.10)
• https://github.com/spesmilo/electrumx/commit/09783aeacc3bae679fa599aae14d9bba23260d63 (ElectrumX; 0.9)
• https://github.com/spesmilo/electrumx/commit/fec2ee1d8f2590a2ef5c970326e378415b4f6d6e (ElectrumX; 1.0)
• https://github.com/spesmilo/electrumx/commit/7d9ffbbed001e512222b57f2161bb087b6b7e86c (ElectrumX; 1.1)
• https://github.com/spesmilo/electrumx/commit/98ca2adc5e38c73738e2ff6a2838fced130729d8 (ElectrumX; 1.2)
• https://github.com/spesmilo/electrumx/commit/9bc102f34a2811a2702995e3b3181568d3e53ff2 (ElectrumX; 1.4)
• https://github.com/spesmilo/electrumx/commit/e3e915323c77cc22ea565993a3b8aeea85c84b31 (ElectrumX; 1.2)
• https://github.com/spesmilo/electrumx/commit/f89cea536c475267ddf869eed9b25fb300414b54 (ElectrumX; 1.4)

Protocol Maximum:

• https://github.com/spesmilo/electrumx/commit/151da40d5b6be87bb95b7c8641262138f0333163 (ElectrumX; 1.0)
• https://github.com/spesmilo/electrumx/commit/135ab68f7419b36d4d565eb47c254ac3240831bd (ElectrumX; 1.1)
• https://github.com/spesmilo/electrumx/commit/2d7403f2efed7e8f33c5cb93e2cd9144415cbb9f (ElectrumX; 1.2)
• https://github.com/spesmilo/electrumx/commit/c35ac6d9bb1df107e09ee5acf25ef50a96bea8cd (ElectrumX; 1.3)
• https://github.com/spesmilo/electrumx/commit/46d8e0c6086f87787fbc35eb329d10ba74e7cc58 (ElectrumX; 1.4)
• https://github.com/spesmilo/electrumx/commit/936251aa960eac928729ba78b5ac11e64a958948 (ElectrumX; 1.4.1)
• https://github.com/spesmilo/electrumx/commit/39817eec7e05c701c0760aeae37aa4dd568e005d (ElectrumX; 1.4.2)
• https://github.com/spesmilo/electrumx/commit/714058794a69ec56563009509a1d6df54ff482f2 (ElectrumX; 1.4.3)

[murchandamus]

Hey @ben221199, this proposal has been emitting a lot of notifications recently without actually requiring editor input. I’m going to temporarily unsubscribe from this pull request while the document is taking shape. Please feel free to mention my handle when it is ready for an editor review. In the future, please consider opening early drafts of proposals against a personal fork of the BIPs repository instead of against the main BIPs repository.

[ben221199]

Hey, @murchandamus. Thank you for letting me know. I will mention you when the draft is done.

[murchandamus]

Hi @ben221199, how is this proposal going? Is there something we can help with?

[ben221199]

Hi @murchandamus, thank you for your message. I had to focus on some other things for my own company the last months, but I'm planning to resume working on this in Q2.

[jonatack]

I'm planning to resume working on this in Q2.

Hi @ben221199, are you still working on this BIP draft?

[ben221199]

Hi @jonatack,

Yes, but I got delayed by some things. With the documentation I wrote for LBRY lately, I think I can pick up this again too. Maybe I start right away.

[ben221199]

Okay, I did read the whole document. I think I'm done when the following things are finished:

• [x] Service server
• [x] Service server.peers
• [x] Service blockchain
• [x] Service blockchain.address
• [x] Service blockchain.block
• [x] Service blockchain.headers
• [x] Service blockchain.numblocks
• [x] Service blockchain.scripthash
• [x] Service blockchain.transaction
• [x] Service blockchain.utxo
• [x] Service mempool
• [x] Explaination of signed messages

[murchandamus]

Sounds good, thanks for the update.

[ben221199]

Okay, I think I'm done.

[ben221199]

I want to ask people to take a look at this PR and come with as much feedback as possible, so that I can add it in before it gets merged.

[ben221199]

Thanks for the feedback. I will try to add most things in, but I hope @slush0 can indeed tell more about the licensing.