Feeds: arbitrary payload instead of arbitrary data structure

[nugaon]

User story

Story

As I noticed, the payload of Feeds is an arbitrary data structure currently with only the time of the upload and the reference of a Content Addressed Chunk. I think it is completely a waste of available chunk payload (in case of passing unencrypted reference, it uses 40 bytes from <4096 bytes). Secondly, what if I want to add to this update more than one reference of swarm content? Or I want to attach metadata as well? This rigid data structure does not let me do that. Seems neccessary to make a decision on a more flexible data structure first. My suggestion is to use BSON instead of this current one, which could be ideal for our use-case (bee-js).

Acceptance criteria

On feed update it is possible to pass arbitrary data that is retrievable later. It could make place for other specific use-cases as well, like dynamic content references in dApp environment (e.g. HTML link references, etc.) and other later. The interface of this payload thereby has to be in generic type, but you still could protocolize for specific use-cases, like this before mentioned one.

Background

This approach proposal came up on the feed interface PR of bee-js.

[Eknir]

What do you believe are the practical use cases of using BSON type of encoding for feeds instead of the current way of doing things?

[nugaon]

Hey! The point is it can open many unconsidered doors which are handled within application layer. Let's suppose I only would like to open the referenced chunk if the metadata in the feed chunk meets with my requirements (e.g. if it describes the linked content is in format avi with length 1 hour 38 minutes, I don't want to even fetch it, because the theoritical application only supports mkv under 5 minutes). Also there can be a use-case if I want to reference hashes of two independent content (e.g. one provides a Map DB, the other provides the client side to illustrate it visually.) where the two content connect together but not neccesary I want to download/use one side of it (e.g. I just want to download the DB part for my application that can interpret).

So I can go on, the goal of this approach is decrease the number of unneccessary requests while utilize anyhow the remaining unused data of the feed chunk.

[Eknir]

I believe this is a good suggestion, but I see it as a nice to have. A higher priority for the dev team is hardening Bee.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nugaon]

related PR that aims at JSON serialisation in Feeds and also points out the current error message that prevents to support this behaviour in application layer: https://github.com/fairDataSociety/swarm-feeds/pull/2

[acud]

@nugaon feeds in their current form enforce a strict data structure, because that's part of the definition of feeds as a construct. If you want to change the data structure it will not be a feed, but something else. I hope this answers the underlying concern. In theory you don't need to use feeds if they don't fit your purpose, but build some other type of construct that uses soc as the underlying chunk type in order to provide certain guarantees. As we all know, feeds can be entirely implemented outside of bee, so why not do things in the way that suits you best?

[nugaon]

@acud you are right about its logic can be implemented outside of the Bee. Nonetheless, if the Bee cannot interpret these extended payloads as feeds - when these are inteded to only extend the default feed capabilities - then one won't able to fetch feed reference on the BZZ path and it won't be supported by other tools as well that use feed endpoint to fetch feeds. Although, the latter can be sorted out by integrating and replacing the feed endpoint usage with the application layer implementation, the application layer feed lookups cannot perform as well as if it happened on the Bee side; e.g. the browsers have really tight limitation for concurrent requests towards one server (Bee), thereby slows down the lookup time.

[ameer-clara]

Legacy feed swarm implementation had no restriction on the payload. Was there a reason to make it of Reference type? Would love to find out when this feature could be part of bee-js?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[nugaon]

.