Add a 'Binary' builder

[jt-nti]

The current CCaaS builder is great for chaincode development and debugging however it is not simple enough when trying out Fabric for the first time. The old built in docker-in-docker chaincode support is superficially simpler but often causes problems when the chaincode fails to build, which can be hard to debug.

While a simple binary builder is unlikely to be suitable in a production environment, it would provide a more robust mechanism to deploy chaincode out of the box and give a better first impression. It may also make it possible to deprecate the old docker-in-docker approach in v2.5 and remove that mechanism completely in v3, although that would also depend on the ability to package Binary (and CCaaS) chaincode.

cc @denyeart @mbwhite

[jt-nti]

It might be nice to follow a similar pattern to the k8s builder and include details of the chaincode binary in a binary.json file (or a better name?!) rather than the actual binary. For example,

{
  "chaincode": "file:///path/to/binary",
  "checksum": "27871baa490f3401414ad793fba49086f6c855b1c584385ed7771e1204c7e179"
}

That would potentially allow chaincode to be downloaded by the builder using https://, either with an initial implementation or as a future enhancement.

[jt-nti]

Worth mentioning that all the peers in the same organisation would need to be on the same architecture to install the chaincode, and they would not be able to move to a new architecture. The intention for this builder is to improve the first use of Fabric, rather than for production networks, so that should be ok.

[mbwhite]

Possibility to go even further and add a signature verification of the binary maybe? But for development that might well be excessive - a checksum at minimum though.

So this would be a single checksum per organization...

[jt-nti]

Signature verification would be nice to have in a production environment but it's possibly excessive for this use case. That could certainly be included if the demand was there though.

There could potentially be a single checksum for all organisations if they share the chaincode implementation, but certainly no more than one checksum per organisation.

[denyeart]

This sounds reasonable to me, as a way to make building and deploying chaincode simpler for people trying out Fabric, while removing the docker-in-docker dependence.

[jt-nti]

There's now a sample binary builder for Go and Node.js chaincode in the nano test network, which is similar to the existing builders in fabric integration/externalbuilders I think. Those also still build the binaries during the Fabric lifecycle rather than using prebuilt binaries though.

See hyperledger/fabric-samples#1215

[arne-fuchs]

So what are the required steps to accomplish that? It might just be simple as defining a custom builder to execute those binaries and put it in the core.yml?

[jt-nti]

@arne-fuchs, I think the first step is to agree how the builder should work. For example:

• Should the chaincode package contain binaries artifacts directly, or just sufficiently identify binaries available somewhere else, e.g. the peer file system, or an external artifact repository?
• How will the builder handle Java, Node and other chaincode that doesn't end up being a single executable binary?
• How will the builder handle different platforms/architectures?
• Should the builder be supplied out of the box like CCAAS?

Maybe it's worth adding something to the maintainers call agenda if those still happen @denyeart?

Once that's done, we should have a better idea of what will end up in code.tar.gz, for example a binary.json and/or a launcher with directories for various platforms/architectures, and what the phases of the builder actually need to do.

As for implementation, assuming a binary builder is provided along with the CCAAS builder, there should be some shared function with the existing builder.

These might also be useful for reference:

• Binary builders from the integration tests
• Binary builders from the nano test network
• The k8s builder

[arne-fuchs]

@jt-nti

Should the chaincode package contain binaries artifacts directly, or just sufficiently identify binaries available somewhere else, e.g. the peer file system, or an external artifact repository?

In my current external builder implementation is, that if you give the type "binary" in the metadata.json, it will look for a binary called 'chaincode' which it will just execute. This, of course, will only accept one architecture. But, in my opinion, it is important to keep the binary packaged so it can be distributed easily.

How will the builder handle different platforms/architectures?

The question is, how the package id is being calculated when installing a chaincode. If it takes the whole code.tar.gz, it is possible to include different binaries for different architecture which can be executed by the builder, depending on the platform. The binaries can be put into the respective folder into the code.tar.gz. x86/chaincode would be on x86 architecture and arm64/chaincode for ARM based systems, respectfully.

Should the builder be supplied out of the box like CCAAS?

Being able to execute binaries is not a highly complex feature imo and might be expected by many organizations using the network. Having it preinstalled probably just increases the working-out-of-the-box-experience with fabric.

How will the builder handle Java, Node and other chaincode that doesn't end up being a single executable binary?

My first two approaches would be these:

1. The metadata type will be extended to something like 'java-binary' and another builder, besides the binary builder, executes the jar with a preinstalled java runtime. Analog to other languages. Of course, the problem is, that fabric needs to maintain the runtime versions by themselves. E.g. when to upgrade the runtime, which may could lead to java code being broken.
2. We shift the runtimes into the code.tar.gz which requires architectural differentiation. So just like for the binaries above. And then we might want to define the runtimes besides the binaries for housekeeping, so you might want to have an architecture like this:

code.tar.gz

• runtime
• • x86/java
• • arm64/java
• chaincode.java

This might lead to larger chaincode packages but shifts the responsibility to the chaincode developer. Parameters for the runtime might be provided in the metadata.json In my current, own written external builder for rust, I made the additional metadata.json field to define the name of the binary so it is able to find the compiled binary. This might of course, be automatically detectable but for developing purposes it is enough, but just want to say that it is always possible to add optional stuff in the metadata.json

3. Of course there is also always the option to not support "binaries" if the require a runtime. At least, they are not really a binary though and would add a lot of work besides the minimalistic binaries.

So these are just basic ideas for approaches. Own thoughts or feedback would be appreciated.

[jt-nti]

But, in my opinion, it is important to keep the binary packaged so it can be distributed easily.

That makes sense. After another fabric samples builder related question on discord, I was also thinking keeping the builder as simple as possible might be the way to go, e.g. not even worrying about supporting different architectures. That way, there should also be less confusion about whether a binary builder would be suitable for production environments.

Having it preinstalled probably just increases the working-out-of-the-box-experience with fabric.

Agreed, increasing the chances of a working-out-of-the-box-experience with Fabric is the main advantage of a binary builder, so not including it by default would be odd!

it is always possible to add optional stuff in the metadata.json

I'm a bit nervous about adding things to metadata.json rather than following the CCAAS approach of a builder specific json file, although maybe I'm being paranoid.

So, based on your ideas, does something like this sound like a good starting point?

• the binary artifacts are inside code.tar.gz.
• the command to run is specified somewhere, e.g. a binary.json file, which could then handle node scripts, a java cli, or whatever. (Checking for runtimes, e.g. a suitable JVM, would be handled by this command.)
• the platform the package targets is is specified somewhere, e.g. a binary.json file, which the builder can use to fail with a clear error message if the peer platform does not match.
• the builder would be provided/pre-configured in the same way as the CCAAS builder.

[arne-fuchs]

In my opinion it is less overhead if we'd put the metadata into the existing metadata.json instead of creating an additional file for the binary metadata. As far as I understood, the metadata.json is exactly for use cases like that and the node does handle it well.

[jt-nti]

There are definitely advantages to using the metadata.json file directly. I'm not 100% sure the builder has access to it at every step, although that shouldn't be a problem.

I guess the main reason that I'm hesitant is that the documentation isn't totally clear whether the file is exactly for use cases like this. The current CCAAS builder suggests that it is not, given that CCAAS packages require a separate connection.json file.

On the other hand, the doc mentions a Go specific path key, which suggests that metadata.json is expected to contain builder-specific information.

There's no problem with keys clashing between builders, because only one builder will ever detect the chaincode package, but there is a vanishingly small chance that the external builders and launchers feature could introduce new conflicting keys in the future.

It would be good to set the right precedent (and maybe update the doc along the way) so this might be a question for the a maintainer call to get consensus (excuse the pun!). Maybe reserving a "chaincodeData" (or some better name) key would work? Something like...

{
    "type": "binary",
    "label": "fabcar",
    "chaincodeData": {
        "platform": "linux/amd64",
        "run": "cc.sh"
    }
}

[arne-fuchs]

Yes I agree that a maintainer call would be the best. I already have build a slim binary executer (and one with cargo for rust) and the metadata.json is fairly available through the building steps, o that should be no problem. And as you said, the path key is already used in the container use case, so why not adding a key for the binary stuff too, right?

Maybe reserving a "chaincodeData" (or some better name) key would work?

Yes, good idea. In your suggestion the architecture specific paths of the binaries would be lifted in the metadata.json, which is also a idea, instead of my directory approach.

Definitely need more consensus on that topic in the next maintainer call. Where can I find the calendar for the next call?

[jt-nti]

It doesn't look like there have been any for a while but the next one is meant to be Wednesday 17 December... https://wiki.hyperledger.org/display/fabric/Contributor+Meetings

I'll ask on the Discord channel.