sydent
sydent copied to clipboard
Remove the v1 API
Blocked by matrix-org/synapse#9677
The deprecated v1 identity service API is a source of security issues and poor UX. We should set a date to remove it, and follow through with that.
Some notable places this is still used:
- Synapse (see https://github.com/matrix-org/synapse/issues/9677)
- Element-web (for .well-known validation or something - see MSC2499, https://github.com/matrix-org/matrix-spec/issues/557)
- we see a lot of requests from mxisd user-agents. mxisd is unmaintained aiui, so it's hard to know how best to proceed here.
mxisd is more likely to be ma1sd: https://github.com/ma1uta/ma1sd
Would be good to set a public deprecation timeline for this
related: https://github.com/matrix-org/matrix-doc/pull/2713
@turt2live suggests that there is other work that might need to be done before we can remove the v1 APIs, but he'd need to go and spend some time figuring out what that is. (I hope that is a fair representation)
Notice that email invites are still using V1 api in order to sign the invitation.
Also we cannot just change the email template to use the v2 API, because v2 API requires consent for the sign-ed25519
endpoint, this will lead to a pretty bad invite flow.
This API is defined as helper for client that cannot do crypto, but still element clients are using it. I tried to get more info on what clients are supposed to do but failed to find more info.
I guess that before removing this API we should:
- Update documentation to explain the invitation signing process (for client to do it without server)
- Update EW, EA, EI to do the signing themself from the invite link
- Modify the email template to not send a sign URL (as it wont be usable as is because in v2 params are in request body and not in query) but instead send the token, private_key as well as the identity server url as regular parameters
I decided to try and pull some figures on where the v1 API is being used. Over the 7 days to 24 June:
- Lookup API:
- 53719 calls to
/v1/lookup
(excluding those from our internal Prometheus :roll_eyes:). These are split:- 52538: mxisd (https://github.com/ma1uta/ma1sd/issues/114)
- 1129: Synapse (https://github.com/matrix-org/synapse/issues/13206)
- 10195 bulk lookups via
/v1/bulk_lookup
. Useragent splits:- 9602 Riot.im/0.9.1 on Android. Suspect this is a single user doing something weird
- 435 Riot on iPhone, with weird version numbers. Forks/old versions? None from current versions of Riot-iOS.
- 124 AOCIM/1.5. No real idea what this is.
- 53719 calls to
- Due to https://github.com/matrix-org/synapse/issues/5881:
- 6082
GET /_matrix/identity/api/v1/3pid/getValidated3pid
(of which 468 were successful: the remainder were polls) - 200
POST /_matrix/identity/api/v1/validate/email/requestToken
- 448
GET /_matrix/identity/api/v1/validate/email/submitToken
- 604
POST /_matrix/identity/api/v1/validate/msisdn/requestToken
- 405
POST /_matrix/identity/api/v1/validate/msisdn/submitToken
- 6082
- Due to https://github.com/matrix-org/synapse/issues/13171:
- 78
POST /_matrix/identity/api/v1/3pid/unbind
- 78
- Due to https://github.com/matrix-org/synapse/issues/13206:
- the 1129 calls to
/v1/lookup
mentioned above - 1193 calls to
POST /_matrix/identity/api/v1/store-invite
from Synapse - 13
GET /_matrix/identity/api/v1/pubkey/isvalid
- the 1129 calls to
- Unclassified:
365 POST /_matrix/identity/api/v1/sign-ed25519 17 GET /_matrix/identity/api/v1/sign-ed25519 288 GET /_matrix/identity/api/v1/pubkey/ephemeral/isvalid 8 HEAD /_matrix/identity/api/v1/validate/email/submitToken 6 OPTIONS /_matrix/identity/api/v1/account/register 3 GET /_matrix/identity/api/v1/terms 3 GET /_matrix/identity/api/v1/pubkey/ed25519:0 3 GET /_matrix/identity/api/v1/hash_details
/_matrix/identity/api/v1/sign-ed25519
is a strange one.
3PID (email) invites include a link like https://app.element.io/#/room/%21cCpRBTFuXwPmOCUFRL%3Amatrix.org?email=michaelt%40element.io&signurl=https%3A%2F%2Fvector.im%2F_matrix%2Fidentity%2Fapi%2Fv1%2Fsign-ed25519%3Ftoken%3DcBeQOefWJXQrcgHavWuqjMtiazBrYMCvLbpGvOSWFLkitbMbuFGyEmgBbcPtfHhHxiKKsdEbEXSJezCbTXowCuFaQIKV%26private_key%3DRkzs-2BDCjkOOnS74LwLRO_Uw&room_name=&room_avatar_url=&inviter_name=Michael%20Telatynski&guest_access_token=&guest_user_id=&room_type=
Which passes a signurl
using the aforementioned API, but encodes the params in query params rather than a POST body which the v2 API docs state the API demands. This means a simple v1->v2 port isn't so simple as now clients would need to receive the parameters outside of the URL and include them in a POST body themselves, with no backwards compatibility.
The API description being
To aid clients who may not be able to perform crypto themselves, the identity server offers some crypto functionality to help in accepting invitations. This is less secure than the client doing it itself, but may be useful where this isn’t possible.
when in reality clients can't really do the signing themselves without parsing the signurl in brittle ways given the URL params don't match any known documentation means that clients are forced to blindly just call a given URL in undocumented ways.
For links, the endpoint in question used to be specced at https://matrix.org/docs/spec/identity_service/r0.3.0.html#deprecated-post-matrix-identity-api-v1-sign-ed25519; note that a GET
variant was never part of the spec.
@t3chguy (and @BillCarsonFr, since I think you said the same thing): I'm failing to grok how this "signurl" is used in the 3pid flow. How is the URL generated? How is it used? I can't find it mentioned at https://spec.matrix.org/v1.5/client-server-api/#server-behaviour-7.
It is being used as POST but with no body and all params being query params, so still doesn't match that spec
The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code
The client includes a third_party_signed
in the /join/
request which is the object it got back from the sign-ed25519
API, the path & params to which it got via signurl
in the link in the email template above
It is being used as POST but with no body and all params being query params, so still doesn't match that spec
Right. Yes.
The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code
Is that the link you meant? It doesn't seem to give anything useful here.
Yup - all the templates have v1 references to the API.
that link definitely doesn't show me those results, but thanks
Ah, might be because I'm on the Github Search Beta which makes the search quite usable
I'm failing to grok how this "signurl" is used in the 3pid flow. How is the URL generated? How is it used? I can't find it mentioned at https://spec.matrix.org/v1.5/client-server-api/#server-behaviour-7.
@richvdh worth noting that there's two flows an email invite can take, and only one of those flows is documented in the spec. The other is completely missing (I couldn't find an issue at first glance, but know this to be true)
@richvdh worth noting that there's two flows an email invite can take, and only one of those flows is documented in the spec. The other is completely missing (I couldn't find an issue at first glance, but know this to be true)
Opened https://github.com/matrix-org/matrix-spec/issues/1359 to track this, then.
Re the signurl
parameter: based on the evidence of https://github.com/matrix-org/sydent/blob/main/res/matrix-org/invite_template.eml and friends, it looks like:
-
The
sign-ed25519
URI is a sydent URI generated by sydent, and as such we can consider it to be an implementation detail of sydent, and there is no need for it to be part of the spec.Following the example of similar endpoints in Synapse (such as the SSO callback), ideally it would not be on the
/_matrix/...
path and instead would be called something like/_sydent/get_stored_invite
, to emphasise that it's not part of the spec. It would also ideally be documented in the Sydent docs.I have opened https://github.com/matrix-org/sydent/issues/533 to track this.
-
The
signurl
parameter - indeed, the whole of the "single-click" link in the email, including the/#/room/<room_id>
fragment, and the other pseudo-query-params - appears to be somewhat specific to Element, though since homeservers can change the target of the link via the unspeccedorg.matrix.web_client_location
parameter, the whole thing seems somewhat dicey. I have opened https://github.com/matrix-org/matrix-spec/issues/1360 to track this.It has to be said that a good start would be for the format of that link to be documented somewhere - either in the element-web docs, or the sydent docs. But in any case that's entirely orthogonal to the removal of
/v1/
.
and as such we can consider it to be an implementation detail of sydent, and there is no need for it to be part of the spec.
Yes, and no. Element Web code has to append a &mxid=<my_mxid>
to the URL, presumably removing it would make it stop working, this behaviour means we'd need to spec that you need to do such for signurl somewhere.
Element Web code has to append a
&mxid=<my_mxid>
to the URL, presumably removing it would make it stop working
Ugh. But also, surely that would work just as well if the root of signurl was https://matrix.org/_sydent/get_stored_invite?...
instead of https://matrix.org/_matrix/identity/api/v1/sign-ed25519?...
? In other words, the location of the endpoint is entirely internal to sydent, rather than being a matter for the spec, even if the parameters that should be passed to the endpoint have to be specced (which would be part of matrix-org/matrix-spec#1360).
Quite right - especially the ugh
Note that there is a config option to disable V1 bindings: https://github.com/matrix-org/sydent/pull/267 - although currently this is only applied to the bind
endpoint: https://github.com/matrix-org/sydent/blob/c9980a9fb89d7fa00ef45601294312110f47c565/sydent/http/httpserver.py#L129-L130