specs
specs copied to clipboard
ipfs
IPIP-270: Bitswap 1.3.0 - Tokens (and auth) support
This is a spec proposal continuing from #260 and other issues related to support for authentication, payments, etc. in Bitswap. This builds off of #269 which helps clean up the Bitswap spec a bit and bring it up to date.
The main thrust of the proposal is support for adding a set of tokens to requests and responses to allow metadata. These tokens can convey various application specific requirements or interactions
A bonus component added to the proposal is adding a BlockTooBig type of BlockPresence so that proper error messages can be communicated for that use case rather than either causing a stream reset or something more confusing like returning a Have, DontHave, or not responding at all.
This kind of change has been needed for a while, and while this may not resolve everything it seems like it will help us to have a concrete proposal to critique and improve.
cc some folks who have been asking about this for a while: @Stebalien @ianopolous @csuwildcat @obo20 @carsonfarmer @bmann
I'll give some background and pre-empt some questions here. If any of this is interesting enough to put in the spec we can do so there.
Unfortunately, GH is pretty bad at threaded discussions not tied to a particular piece of code so if you have a comment about something not explicitly in the code just grab some empty line in the PR and start a thread there.
Background
In Bitswap versions 1.2.0 and earlier it is not feasible for a server to decide on whether a peer is authorized to receive a particular block of data other than by discriminating on the peer itself. Consider if Alice wants to request a block B from Bob he will serve it to her if she is on his allowlist, and otherwise will not. If Alice wants to gain authorization to download B she can use some external protocol /authproto to negotiate her authorization for B with Bob.
This precludes a number of use cases including:
- Multiple auth mechanisms per peer
- Multitenancy: If Alice was a multitenancy client that was trying to download data from Bob on behalf of Charlie and Dan she doesn't have a good way to know which of Charlie or Dan are supposed to have access to B which means she could be handing over B to Dan even though he shouldn't have been given it in the first place. This might be very useful for semi-trusted intermediaries that are able to download data via Bitswap on behalf of users in more restrictive environments.
- Per Retrieval Payments: Similar issues arise in the case of Alice and Bob trying to negotiate payments for the retrieval of chunks of two DAGs D1 and D2 both of which contain the block B. When Bob sends Alice B if payments occur after receipt then Alice doesn't know if Bob is expecting to be paid the D1 or the D2 payment rate for B. While probably Bob should just charge whichever is lesser, if the payment scheme is more complicated this might be non-obvious or generally determinable
- Using auth as a fallback: i.e. trying to fetch data without auth before falling back to using auth
- If Bob has B and Alice asks for it but is not yet authorized there is no way for Bob to communicate the difference between "I don't have the data" and "I have the data but you need to authorization" if he so chooses. This is particularly problematic in the case where the authorization is open (e.g. anyone who wants to pay for the data, anyone who solves a CAPTCHA, etc.) since then the options are to not believe any DONT_HAVEs and waste resources negotiating authorization or for Alice not be able to download data from any peer she didn't know in advance she needed to authenticate with
Alternatives/FAQs
- Why not remove the
AuthRequiredtoken and instead put auth requirements in the content routing system?- Requires going to the content routing system frequently (potentially for every block) to figure out if authorization is required which defeats the important Bitswap optimization of skipping content routing by asking connected peers if they can send us blocks
- Requires content routing systems to support the auth mechanisms which is a bigger ask than if it could be done at the exchange layer
- Why support arbitrary token types?
- Proposals like #260 that do not differentiate different token types make it difficult for the application layer to know what to do with the tokens it received. It may have to do some testing on the tokens to figure out what they are for. It may also lead to ambiguity when a token could be used for two separate authentication schemes
- IMO a generally useful piece of multiformats is that for the low cost of a few bytes you can figure out what your data is, that seems like something to take advantage of here
- Why not dictate exactly what tokens can/should be used for?
- Mostly because application developers will do whatever they want here anyway unless it explicitly breaks the protocol. We should give guidance, but any MUSTs here are likely pipe-dreams even if we wanted them
- Why is there a level of indirection around tokens and token indices?
- Basically to save bytes since we're not using something like gzip on the messages. It is expected that certain scenarios will have many blocks reuse the same tokens and we can save a lot of bytes there, while not being too costly in the one-token-per-block case
Hi @aschmahmann Thanks for making a start on this and doing the prior spec update!
I would argue that a byte[] is the most general you can get. In fact, for our usage in Peergos these byte[] are cbor. So we get the benefit of types, upgradability etc. but at the application layer and without the centralisation point of multicodecs. And this also doesn't force everyone to pay the overhead of the multicodec. This means applications can innovate faster and without requiring anyone's permission.
The other change I would suggest is to allow per cid auth tokens inline, as we do. Otherwise you're forcing someone like us to pay the extra overhead of the indirection (No auth tokens are ever shared between blocks for us, and arguably that would be a vulnerability (e.g. S3 doesn't allow this)). Probably there is a use case for both, especially if you include non capability-based schemes. It should be easy to support both inline tokens and indirect references.
Do you have a concrete use case in mind for allowing multiple auth tokens for a single cid in the same request? Our usage in Peergos is maximally fine grained and doesn't need this. For example, Amazon S3 doesn't need this and they clearly support almost all use cases in the world right now.
One other nice to have is to maintain backwards compatibility with our prior art coming up with a secure capability based auth scheme for Peergos and the associated bitswap extension. We have just extended the protobuf for bitswap 1.2, and this PR is making a new protocol id 1.3. However at the moment all versions of bitswap share the same protobuf. This is a nice simplification to not have to maintain code around N different versions of the protobuf. I think the simplest thing is just to avoid the field indices we've used in https://github.com/ipfs/specs/issues/260 ? Or even better, extend our protobuf which only adds optional auth strings. This means users can choose between the indirect list of auth tokens, or an inline one.
I think gzipping would be nice, although it'd be nicer if it could be done at the libp2p layer
strong agree, we should push for adding compression in libp2p, not try to add it manually to protocols
Without some form of signaling every time I want to add support for a new token type I have to make sure everything still plays together nicely.
I would argue you have to do that anyway, if you want the systems to actually attribute real meaning, just having cbor in there will not help that much in practice, as the semantic meaning is much more critical than the encoding format.
In general, I expect not to be using cbor at all, especially in protocols that are based on protobufs, so adding a requirement in here for cbor, would mean requiring another encoding format ontop of the one this is wrapped in, so I am strongly in favor against requiring cbor.
Hi :wave: I'm part of the fission team. This is an interesting proposal for us, since it'd technically make using UCANs possible with bitswap directly.
Thoughts:
- Multiple tokens is a :+1: from me. Technically you could combine tokens into your own custom list format inside the byte array, but in combination with the multicodec proposal I think this way has some advantages, e.g. if tokens get forwarded into the DHT, different nodes may want to pick up different subsets of tokens, and we'd want them to agree on the format for "finding the subset".
- Multicodecs for tokens is also a :+1:. Multicodecs are the most minimal signalling protocol I know and they're used throughout the IPFS stack. I think the biggest value of this is forcing applications to use some signalling in the most minimal way. The worst possible situation would be allowing any bytes and someone ending up using a token where all byte sequences have a valid meaning.
- I'm undecided on whether to remove the "custom compression" and just go gzip on another level. I'm generally in favor of simplifying protocols, however in this case I think the complexity is really low and it's going to be a very common case of re-using tokens for multiple CIDs, if not the very default case. The repeated bytes will have to be represented at some point in memory, even if we're streaming them into gzip, the stream will need to repeat the token. Just wanted to note this, it may still be premature optimization, as I said, I'm not quite decided on that yet.
In any case, I can see this making UCAN use-cases possible with bitswap which is exciting for fission.
@matheus23 Just a warning that using a UCAN in the token field here would also be totally insecure (in normal ipfs bitswap usage) as per the discussion above unless the audience of the ucan is the node id of the requesting ipfs node AND this is verified on the receiving end against the actual requestor, in addition to the usual ucan checks.
Just a warning that using a UCAN in the token field here would also be totally insecure (in normal ipfs bitswap usage) as per the discussion above unless the audience of the ucan is the node id of the requesting ipfs node AND this is verified on the receiving end against the actual requestor, in addition to the usual ucan checks.
Thanks for making that clear. We're aware of that. :+1: Sorry, I'm trying to read between the lines here... Would you expect my stance to change given that information?
Status Update:
Long overdue status update from the implementers sync (https://pl-strflt.notion.site/IPFS-Implementers-Sync-2022-05-12-b338b20567904923b3772c58f1c9186c) here is that this spec PR needs to be updated but should otherwise be reasonably solid. IIUC the remaining points were 1) smaller fixes to text in the spec 2) changing the protobuf numbers to skip the ones used by Peergos.
Ideally I'd like us to have two users of this work to help prove out the spec change's utility. Peergos is one use case that's also not blocked since they currently have an alternative mechanism riding on top of Bitswap 1.2. I was previously leaning on some of the folks working on payments to provide that second user story, but the folks working on Boost have delayed that work.
@matheus23 is this something Fission is interested in working on/using? From the Bitswap implementations I've looked at the server side changes to support tokens are fairly minimal, but the client side changes (as Ian has mentioned) require more work to fit the use case and consider how clients figure out which peers to send a given token to.
Otherwise you are trivially vulnerable to replay attacks because you broadcast the tokens to the DHT
We've done some work with UCAN to avoid replays, too 👍 The most general solution is to enforce token uniqueness (via a nonce), so that replays are impossible (under normal assumptions: the protocol is adhered to, etc).
unless the audience of the ucan is the node id of the requesting ipfs node AND this is verified on the receiving end against the actual requestor,
@ianopolous I think some of the roles are swapped, but yes that's how UCANs must be used in all circumstances. The issuers and audience need to form a chain all the way back to the original node, including the node that is fulfilling the request.
For what it's worth, putting a UCAN right in a bitswap header probably doesn't make sense due to its size. As alluded to earlier, in point-to-point sessions, it's possible to do a handshake with a "blank"/self-targetted UCAN, and then send the triple aud, ucan_cid, nonce to shrink the size of the payload. We've also started playing around with the idea of a hash OTP, which seems to work well.
cc some folks who have been asking about this for a while
Maybe to clarify a bit: the Fission team definitely welcomes this as a privacy measure, and I can see this potentially enabling more generic protocols for interacting with service providers, creating ad hoc private networks, etc. Given the often long lead times for changes in libp2p to get into projects like IPFS, working on this now seems like a good idea 💯
That said, Fission doesn't have a burning need for these features today. I like the idea conceptually, but we probably won't go out of our way to use it until it's more baked in.
I also wonder if it would be possible to move this kind of a spec up a level from Bitswap? We're starting to explore alternatives to Bitswap, and having a generic authenticated session in libp2p is attractive.
Otherwise you are trivially vulnerable to replay attacks because you broadcast the tokens to the DHT
We've done some work with UCAN to avoid replays, too +1 The most general solution is to enforce token uniqueness (via a nonce), so that replays are impossible (under normal assumptions: the protocol is adhered to, etc).
@expede Unfortunately, this is still broken by what I mentioned above - it reduces the replay to a (trivially exploitable) race condition. Nonces don't make it secure.
unless the audience of the ucan is the node id of the requesting ipfs node AND this is verified on the receiving end against the actual requestor,
@ianopolous I think some of the roles are swapped, but yes that's how UCANs must be used in all circumstances. The issuers and audience need to form a chain all the way back to the original node, including the node that is fulfilling the request.
Based on my reading of the ucan spec the terms are correct. What I'm saying is in addition to a being a normal well formed ucan.
That said, Fission doesn't have a burning need for these features today. I like the idea conceptually, but we probably won't go out of our way to use it until it's more baked in.
Could you clarify what you mean by "more baked in"? I want to double check that you're not waiting on go-ipfs (now kubo) to implement something like ipfs get bafyfoobar --token=<some-token> since that's not something that's planned to be implemented for a combination of the UX and security reasons that have been mentioned above. However, in the case of other implementations and applications, supporting tokens like this makes things like downloading data authenticated with tokens much easier (https://github.com/ipfs/specs/pull/270#discussion_r857955469).
I also wonder if it would be possible to move this kind of a spec up a level from Bitswap? We're starting to explore alternatives to Bitswap, and having a generic authenticated session in libp2p is attractive.
Having a generic authenticated session in libp2p seems like an interesting idea, although I don't think that solves our problems here since now if there are multiple possible tokens to use to download a given block of data (or multiple tokens that must be used together) life becomes much more complicated.
If you have a proposal in mind that seems like a great IPIP (https://github.com/ipfs/specs/pull/289), or a libp2p spec proposal. As you may have seen, the suggestion for a compression libp2p protocol that other protocols could layer on top of has come up which seems like it would need similar mechanisms, and is an area I think would be pretty cool for people to explore. However, it both seems like more work than this proposal and so far seems like it'd be less efficient as well so I think this proposal as-is currently makes more sense.
it reduces the replay to a (trivially exploitable) race condition
To be clear, one absolutely critical property of any usage of this is that auth tokens MUST be tied to the sender's nodeId and this must be verified before releasing a block.
@ianopolous you're going to have to walk me through that race condition. They can't connect as a node in the DID chain. This is an extremely limited number of peers, all of who are granted access by transitive closure. It ties them to the nodeId in the chain.
I want to double check that you're not waiting on go-ipfs (now https://github.com/ipfs/go-ipfs/issues/8959#issuecomment-1156963447) to implement something like ipfs get bafyfoobar --token=
since that's not something that's planned to be implemented for a combination of the UX and security reasons that have been mentioned above. However, in the case of other implementations and applications, supporting tokens like this makes things like downloading data authenticated with tokens much easier (https://github.com/ipfs/specs/pull/270#discussion_r857955469).
@aschmahmann yup, that's kind of the idea. I was just downplaying my own comments as possibly not being relevant since if it doesn't land in Kubo or Iroh, we likely won't use this feature in the near-to-medium term, but I like the general concept. Boris was pinged on the issue, so we wanted to make sure that Fission weighed in!
We've had some inbound interest in using UCAN for these kinds of use cases, so there's some experimentation happening with that at other layers. They're probably too big to go with every request, without some extra protocol on top to reduce that size in a session.
@ianopolous you're going to have to walk me through that race condition. They can't connect as a node in the DID chain. This is an extremely limited number of peers, all of who are granted access by transitive closure. It ties them to the nodeId in the chain.
@expede We can go through it in person at the IPFS workshop. I suspect you have an additional assumption or two on top of the ucan spec that I'm not assuming.
It would be great to get a summary of what needs to happen to move this from draft to ready for review, since this has been sitting for a few months now and we know there are several implementation interested in making use of this mechanism.
It would be great to get a summary of what needs to happen to move this from draft to ready for review
See https://github.com/ipfs/specs/pull/270#issuecomment-1156612945.
I've fixed up the small changes in the spec, but we're still lacking multiple groups planning to implement and utilize this behavior so that we can make sure that the abstraction here is sufficiently usable for folks. Maybe it's missing something or will be a pain to work with on the implementation, especially on the client side.
As has been noted multiple times in this PR depending on what types of tokens you're planning on passing along things implementations might need to deviate from how they work today, particularly on the client side. IMO merging a spec change to a core protocol while it's not being used is not responsible.
and we know there are several implementation interested in making use of this mechanism.
While there have been multiple groups asking for this functionality in the past, AFAIK Peergos is the only group that has expressed an interest in actually building out the pieces they need to use this and IIUC they're currently fine using their fork of Bitswap 1.2.0 which does enough of the job for them but doesn't really support any other use cases.
Some of the other groups that initially wanted this (e.g. for wanting to support paid retrieval of data served over Bitswap, limiting serving data using some permissioning schemes similar to bearer tokens or UCANs, etc.) have since decided that their need is not yet high enough for them to want to prioritized implementing the changes. If this has changed and we have an additional group or two that want to implement and utilize these changes then it'd be great to see how the implementations go so we can either merge the spec as is, or make any changes that end up being needed for implementers.