ipfs
gateway: Try feather for incremental files verifications in gateway
The goal is to handle efficiently handle links between the left and middle column, for that I'll experiment by porting https://github.com/Jorropo/go-featheripfs to be used in the gateway code (as well as some modernising).
It is pretty unclear how much breaking I'll have to do. The simplest option is to fork the gateway code (not viable long term). Else I can try to find some commun interface without compromising on speed too much (I don't belive in it).
The other (my preferred) option is to drop go-unixfs from the gateway, this will include challenges (ipld compat), but will allows us to use a server driven interface internally (see ipfs/go-libipfs-rapide#12).
Done crateria:
- Have a unixfs client that is is usable to implement the gateway API spec that decodes an IPIP402 stream using a fixed (low) amount of memory for arbitrarily sized files.
By "left and middle" column I assume you're meaning columns 2 and 3 in https://www.notion.so/pl-strflt/Project-Rhea-decentralized-IPFS-gateway-3d5906e7a0d84bea800d5920005dfea6#9d1477c5b8d84790bcbb56c0c569f15a right?
2023-05-18 conversation: @Jorropo is going to update the issue description to be very clear about the done criteria. We should spell out the expected inputs and output. The goal is to avoid someone at code review being surprised about what we're doing.