BeeJS modularisation

[nugaon]

In the upcoming period we would like to split the bee-js library into several reusable modules that give more freedom over swarm primitives. Currently the bee-js mostly does API wrapping of the go Bee client. This functionality will resign in bee-js-client with no extra logic around that. Additional packages are the following:

• @ethersphere/swarm-utils: Contains common interfaces and utility functions that we use across our projects like byte operations, base types and assert functions.
• @ethersphere/chunker: Provides functionalities related to chunks e.g. BMT hashing and its interfaces.
• @ethersphere/soc: Single Owner Chunk in memory representation with corresponding utility functions for constructing and mutating this primitive: signer/identifier functions and validating ownership.
• @etherpshere/feeds: Swarm Feeds that rely on the soc library and make flexible structure for easily defining new feed types. The interface for constructing such a feed type will await for custom indexing schema and its corresponding lookup algorithm. With that, you will be able to optimize feed lookups with cashing on client-side or even creating totally new type of feed structures for different use-cases. Of course, it also ships its own primitive that allows different feeds be used in a unified way in other swarm projects.
• @ethersphere/mantaray: we have currently the mantaray-js library but we would like to rename it and use it with the @ethersphere prefix.
• @ethersphere/bee-js: as mentioned above the current API wrapping will be extracted, but bee-js will not disappear. all above mentioned modules will be exposed from this library in a way that javascript dApp developers firmly onboard to the Swarm ecosystem, but still use the power of most of the introduced features. Some planned features here are:
  • manifests functionalities that is based on mantaray primitive but provides high-level CRUD operations
  • working on feeds with JSON objects
  • chunk streaming upload

[agazso]

Just a minor note regarding naming: currently the bee-js package name is @ethersphere/bee-js which is consistent with the github repo name. However this may not necessarily be possible with some other proposed package names, because there are already go projects with those names. For example there is already a feeds repo under https://github.com/ethersphere/feeds.

It would be good to come up with a solution that can provide consistent naming without introducing clashing names.

[nugaon]

great notice, my idea was append the package names with trailing -js.

Any other idea?

[r0qs]

Hey this is great news! And I like the originally proposed names (without the -js). About the feeds repo, I believe that the existent one pointed by @agazso looks more like a poc around feeds, and maybe could be renamed to give room to the other. But anyway, it is very good to have those modules as independent packages, it will bring much more flexibility to dapps developers.

[AuHau]

Also, another thing is to consider how the multi-package setup will be created. The obvious solution that I think everyone has in mind at the moment is to create for each package separate repository. This might have some disadvantages though. Usually, with this setup, the problem is that when you start to work on changes that spawn across multiple packages it is hard to coordinate (you have to release and then propagate the changes), and sometimes even development is quite challenging (you have to use heavily npm link, which sometimes fell short). Another potential problem is the Contribution barrier that this setup creates, where if somebody wants to contribute he has to clone/fork multiple packages and then again coordinate with the maintainers their work.

Another solution is going with monorepo and a system like lerna heaving all the packages as part of one repo (I guess the bee-js repo could be the place), which then makes it easier for cross-package development work. Lerna supports different versioning strategy so we could also decide on that.

I remember people around js-ipfs dealing with this problem in the way of adopting the Lerna monorepo, so maybe we could learn from their work and directly adopt it? I am not saying necessary for all JS projects/repos, but for this bee-js modularization, it would make sense IMO.

Related issue in js-ipfs: https://github.com/ipfs/js-ipfs/issues/2877

[r0qs]

Hi, is there any progress on this? We are interested in using the @ethersphere/chunker but I couldn't find it. Specifically, we are currently using swarmhash to generate swarm hashes for the Solidity compiler binaries (https://github.com/ethereum/solc-bin/blob/6fe4cced82cf0e5149bcf6537b443e2cfafc43a4/update.mjs#L158). However, this library has become outdated since Swarm migrated to the BMT hash scheme, resulting in incorrect hashes being generated for some time now (please see: https://github.com/ethereum/solidity/issues/14700, https://github.com/ethereum/solidity/issues/4092 and https://github.com/axic/swarmhash/issues/1).

We need to replace swarmhash due to its dependency on node version 16, which is coming to its end-of-life. Therefore, to continue supporting swarm-compatible hashes for our release binaries, we're eager to know if there are any plans to expose the low-level chunk functionalities in bee-js.

I believe we could replace our current use of swarmhash by bmtHash or even makeContentAddressedChunk.

However, neither of those functions are exposed in BeeJs. Consequently, we find ourselves unable to utilize it as a replacement.