operations
operations copied to clipboard
openstreetmap
Deprecation of HTTP Basic Auth and OAuth 1.0a
This is to track the discussion of the future deprecation of HTTP Basic Auth and OAuth 1.0a.
See pnorman's diary and https://github.com/zerebubuth/openstreetmap-cgimap/issues/286
See the deprecation announcement for dates and details:
This change will have three key dates:
- March 1st, 2024: New OAuth 1.0a application registrations are disabled. Existing applications will not be impacted. HTTP Basic Auth will not be impacted.
- May 1st, 2024: Sysadmins will start brownouts to find applications that are still using OAuth 1.0a or HTTP Basic Auth
- June 1st, 2024: OAuth 1.0a and HTTP Basic Auth will be shut down.
Over the last week 5% of API calls were HTTP Basic Auth, 23% OAuth 1.0a, and 73% OAuth 2.0. There's some daily jobs that are being run that call OAuth 1.0a starting at 01:15 UTC, 02:00 UTC, and one that calls HTTP Basic at 02:30 UTC, but even excluding those it's 18% OAuth 1.0a.
5 months ago, it was 70% OAuth 2.0, some traffic is moving to OAuth 2.0 but slowly.
I think it would be helpful to have some stats highlighting the top OAuth 1.0a users. This way, we could raise follow up issue in the respective app repo. This would also help us to better track the overall progress.
The number of valid tokens per app id might be a good starting point:
SELECT app.id AS app_id,
app.name AS app_name,
count(tk.id) AS token_count
FROM client_applications AS app
LEFT JOIN oauth_tokens AS tk ON app.id = tk.client_application_id
AND tk.invalidated_at IS NULL
GROUP BY app.id,
app.name
ORDER BY token_count DESC
LIMIT 100;
Possibly there are better ways to gather relevant insights on per-app OAuth 1.0a usage.
I doubt the token count is very useful because some apps save a token and use it persistently and some generate a new one on every use.
More useful might be the number of users with at least one token for an app though there is still the question of how recent it is.
Here's a complete list of UAs that used OAuth 1 yesterday, with a count of requests:
1 FitoTrack
2 CityZen
2 Zend_Http_Client
5 AtlasNavi
5 viking
6 Go-http-client
14 Authlib
17 maps.me
20 Node
22 PHPoAuthLib
30 FireYak
32 Locus
43 OAuth
50 OsmAnd
57 MapRoulette
73 Apache-HttpClient
249 Python-urllib
255 Python-httplib2
301 GuzzleHttp
338 PECL-OAuth
370 osm-wikidata
431 NewOSMContributorsInBelgium,
450 Organic%20Maps
1573 MAPS.ME
2776 Vespucci
3875 okhttp
6254 Dalvik
10015 StreetComplete_ee
13827 python-requests
17775 -
45333 NewOSMContributorsInRussia,
70304 Mozilla
74752 Java
87722 StreetComplete
129793 JOSM
I've reduced it down to just the UA name to exclude version differences etc.
The number of generic frameworks in that list limits it's usefulness (:angry:) so here's some results from @mmd-osm's query but limited to tokens issued this year to try and exclude historic things, and everything over 1000 tokens as a start:
app_id | app_name | token_count
--------+------------------------------------------+-------------
3998 | Geofabrik Downloads | 198430
750 | uMap | 88382
1896 | Organic Maps | 43020
3277 | HOT Export Tool | 22932
2302 | New MapRoulette | 20081
10793 | StreetComplete | 18758
2985 | How did you contribute to OpenStreetMap? | 17070
9790 | MR MKS | 14896
3817 | Osmcha | 11493
5742 | umap-de | 10960
8524 | Organic Maps | 10710
946 | JOSM - Java OpenStreetMap Editor | 9061
5066 | OsmAnd | 9050
7526 | OSM-Boundaries | 7487
821 | OsmHydrant | 5224
1875 | uMap Framasoft | 5214
8520 | OrganicMaps F-Droid | 4526
9922 | tasking-manager-hyd.geo.apple.com | 4478
3175 | overpass turbo | 3583
3551 | OSM Streak | 3489
4188 | TeachOSM Tasking Manager | 3361
5265 | OpenPlaceReviews | 3226
2100 | KartaView | 3098
7404 | MapComplete | 2888
11156 | DEMR3 | 2773
10210 | Vespucci 2022 | 2621
12240 | OSM Imagery | 2616
1198 | umap.osm.ch | 1941
1161 | Osmose | 1586
1917 | GNOME Maps | 1411
952 | Vespucci | 1376
8490 | mr_mumbai | 1249
3215 | Freemap Slovakia | 1200
6020 | OSM Teams | 1070
I think there's plenty there we can target as a starting point...
129793 JOSM
Do we have a better idea of when OAuth 1.0a/Basic Auth will be officially deprecated? Last I heard, it was still under consideration.
I've had a ticket open in JOSM Trac for hiding Basic Auth/OAuth 1.0a (see #22810) which I had scheduled to revisit in January. I don't (yet) want to remove OAuth 1.0/Basic Auth, since it still works, but I don't mind hiding it from new users if necessary.
I'm inclined to say they're deprecated right now - that's literally the point of this ticket that we're trying to get people to move to OAuth 2 instead.
The question is when we stop supporting them.
I wasn't actually bothered by the JOSM entries because I assumed it was just people that had configured it some time ago and that they could migrate when necessary.
Yes, this ticket starts the deprecation process. We have no date for when we will shut down OAuth 1.0a or HTTP Basic Auth, but earlier stated that we did not plan to shut them off this year. I would expect sometime mid-next year but that depends entirely on how efforts to move people to OAuth 2.0 go.
MapRoulette has moved, see https://github.com/maproulette/maproulette3/issues/1876
I reached out to the TeachOSM and Apple Tasking Managers.
OSM-Boundaries: https://osmboundaries.userecho.com/communities/1/topics/66-move-from-oauth-10a-to-oauth-20-for-osm-logins
I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.
I'm not aware of a repo for How did you contribute to OpenStreetMap? Let's try to ping @pa5cal instead.
Thanks, @mmd-osm ! I will try to change my HDYC webpage to OAuth2 in the next few months. The first tests today looked quite promising.
For all (javascript) devs reading this https://www.npmjs.com/package/osm-auth has been updated to use OAuth2.0
To maybe point out the obvious:
- turning off basic auth is mainly a question of improving security / control on osm.org. 3rd party installs will likely continue to use it and so support for it will need to remain in the rails-port.
- turning off OAuth 1a provides a reduction in code and UI complexity for osm.org, but no other benefits. @gravitystorm hasn't expressed himself if he wants to drop the code completely or not, but I suspect there may be again 3rd party installs that will want to continue to use it.
tl;dr a hard requirement for OAuth 2 on osm.org just means that everybody that wants to maintain compatibility with as many installs as possible, that is at least the JOSM devs and myself, simply needs to add additional complexity with no benefits at all.
There are security concerns with OAuth 1 as well - even if the protocol is still good the fact that our implementation relies on unmaintained gems is not great.
tl;dr a hard requirement for OAuth 2 on osm.org just means that everybody that wants to maintain compatibility with as many installs as possible, that is at least the JOSM devs and myself, simply needs to add additional complexity with no benefits at all.
I've been looking at removing support for Basic Auth and OAuth 1.0a from JOSM (see #22810; if we end up going that route, I intend to do an Upcoming changes section in the JOSM Startup notes specifically calling the change out a few months beforehand).
Any OSM server that uses the rails port/cgimap (is there any other?) should support OAuth 2 (OAuth 2 has been supported by the OSM website since June 2021 (>2 years)). If they are using the OSM website stack and they do not support OAuth 2, they should update.
I may reach out to projects that use the OSM website stack and see if they want to give me a set of OAuth2 clientIds to hardcode into JOSM. I'd like to avoid doing this, but a RFC7591 implementation for doorkeeper isn't on the horizon (or in their bug tracker).
@simonpoole : to be honest, I don't have much visibility how people use Vespucci (or JOSM) with 3rd party installs. I can think of OpenHistoricalMap and OpenGeofiction maybe, and possibly some non public in-house use cases. I don't see why we also couldn't give them a heads up. We would need to know who they are, and how to reach out to them.
We should also encourage 3rd party installs to stay somewhat up-to-date with the upstream Rails repo, which would avoid the issue altogether.
I don't have much visibility
To illustrate the issue why it is very difficult/near impossible to know anything about how our systems are used:
There's an Indian project/utility/group of utilities that are using a fork of Vespucci to collect/edit data and likely upload it to a private rails-port instance. I've never communicated with them (and there is no obvious way to do so), the only reason I know they exist is that I've now and then seen crash dumps because they broke things and didn't change the crash dump URL.
For OHM, I pinged them on slack; they've since made an issue: https://github.com/OpenHistoricalMap/issues/issues/610 . They apparently already support OAuth 2.
For OpenGeofiction, we'd probably have to email them -- see https://wiki.opengeofiction.net/index.php/OpenGeofiction:Contact . They may support OAuth 2 already, but I'd have to make an account to check.
For non-public in-house use cases, as noted in https://github.com/openstreetmap/operations/issues/867#issuecomment-1725865973, I was intending to add an Upcoming changes section to the JOSM Startup page specifically to inform users about this. I don't know if Vespucci has a similar "startup news" page that can be updated out-of-band.
I'll go ahead and add the upcoming deprecation/removal to the JOSM startup page now.
We should also encourage 3rd party installs to stay somewhat up-to-date with the upstream Rails repo
AKA update their site at least every 2 years.
EDIT: JOSM startup page edited
I don't know if Vespucci has a similar "startup news" page that can be updated out-of-band.
As a rule we don't phone home.
There's an Indian project/utility/group of utilities that are using a fork of Vespucci to collect/edit data and likely upload it to a private rails-port instance.
@simonpoole : does Vespucci support 3rd party installs using OAuth 1.0a without forking the app? I remember having to do this many years ago for testing purposes, although things may have changed in the meantime.
In general, if someone is running their own stack, I would assume they have sufficient knowledge to handle upstream changes as well.
@simonpoole : does Vespucci support 3rd party installs using OAuth 1.0a without forking the app?
Yes, you can load additional client keys for a while (http://vespucci.io/help/en/17.0.0%20Release%20notes/#improved-key-handling-and-new-oauth-10a-keys). Matter of fact the standard key used now days is configured using that mechanism at build time, we do still have a (different) key in the repo, because F-Droid.
Now if somebody actually uses that is a different question, there is literally no way we can know. But as I pointed out above, the team here needs to decide if it will drop OAuth 1.0a support and then client devs can decide if they want to drop support some time in the future. Which is related to but not the same question as what is supported on osm.org, I'm very much for dropping basic auth support there -now-, but would consider dropping it from the code silly.
As simple as basic auth may seem, it introduced a dependency on Argon2 both on Rails and CGImap, and was quite a pain to get working properly across implementations. In total, we're using 6 different implementations to authenticate users today. The authentication topic has accumulated some technical debt.
we're using 6 different implementations
- ~~Basic Auth~~
- ~~OAuth 1.0a~~
- OAuth 2.0 x2 since we have the rails port and cgimap. Do I have things right? It seems like there should only be one implementation that should matter for each of the oauth methods. I don't think cgimap has to implement anything to do with the authentication dance for the oauth methods anyway, just verify the given tokens. But I don't know if you are counting the verification in cgimap as an "implementation".
Yes, exactly, CGImap needs to validate the Bearer token for OAuth 2.0 only. That's a single database SELECT to check the write_api permission, check for expired / revoked tokens, etc. All the other parts are handled by Doorkeeper on Rails.
Even for Basic auth and OAuth 1.0a, CGImap implements the validating part only, which is surprisingly complex in case of OAuth 1.0a. Overall about 2500 LOC, including test cases.
Could somebody say if, or how, this will affect ordinary end users rather than programmers?
In particular will I still be able to log in to the main OSM pages using my username and password?
- there is absolutely no way that I would be willing to log in using a third party spyware company's account (ie google, facebook etc).
- will I still be able to edit with JOSM, again without having to use a third party spyware site for my login?
- less important, but I occasionally use streetcomplete, overpass turbo and a few other useful sites. Will end users of any of those be affected?
I also have a personal use program running leaflet.js. It uses a few APIs in read-only form:
- the Notes API (which as far as I remember doesn't require me to be logged in to OSM read existing notes)
- overpass turbo queries (again, read only and I'm not aware of it requiring an OSM login
Can I just assume that JOSM and any most other third parties will implement any necessary changes internally and I'll be able to continue doing all of the above with just my OSM username and password?
This is about how things authenticate to OpenStreetMap not about how you login - when you authorise an application with OAuth (whether 1 or 2) you login to OpenStreetMap in the normal way and the app is then given a token it can use to access OpenStreetMap without having to know your password.
I also have a personal use program running leaflet.js. It uses a few APIs in read-only form:
Read access (with the exception of user specific data) currently doesn't require authentication, so yes.
The current load shows that >85% of normal requests come from oauth 2.0, and those coming with OAuth 1.0a and HTTP basic tend to come in batches which indicates some kind of scripted load.