openstreetmap
Get the recently closed / reopened / commented notes
Problem
The API only takes the opening time to get the notes in "Search" option, but I need to get the changes of every note (closed, commented or reopened). The API documentation on the Wiki states this:
Returns the existing notes matching either the initial note text or any of the comments. The notes will be ordered by the date of their last change
The API Search option only returns the notes created after a given date. However, there is no option to retrieve the modifications of the notes after a given date, or the implementation of this function is not working correctly.
Let's suppose these notes:
t2 t1
|------------+---------+----------+-----------------+-------+------+-----+-------------> t
Note1 | | | Open |
| | | |
Note2 Open | | Comment/Close
| |
Note3 Open Close
If I query the notes "from" t1 (to now), I only get Note1. But I also want to retrieve Note2 because there was a comment after t1. The only way to get the Note2 from API is to put the "from" parameter as t2 before the creation of note2, but in this case, I will also get Note3.
Let me show you a scenario on an island in Colombia called Malpelo.
By analysing the Planet dump, I see there are two closed notes:
-
994133 (Closed 12 months ago)
- Created at 2017-05-12 20:13:23
- Closed at 2022-10-17 01:28:35
-
3602820 (Closed one day ago)
- Created at 2023-03-18 14:03:57
- Closed at 2023-10-13
I use the API to get the notes, and I modify the "closed" parameter with a date between the opening and close of the first note, which I chose 2021-01-01:
-
closed=-1 (Error): Returns only the recently closed note but not the old note.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=-1
-
closed=0 (OK): Returns nothing, which is OK. There are no open notes in the area.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=0
-
closed=7 (OK): Returns the recently closed note, which is OK.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=7
-
closed=5000 (Error): Returns the recently closed note but not the other, which was closed 12 months ago.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=5000
-
closed=5000 changed date (OK): Returns the recently closed note but not the other, which was closed 12 months ago.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2017-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=5000
-
closed=400 changed date (OK): Returns the recently closed note but not the other, which was closed 12 months ago.
- https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2017-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=400
Description
The API for the Search option should include a parameter for only open notes or any operation on the notes. It only works for open notes, but the API states the other. Therefore, both options should be part of the API with an extra parameter.
Screenshots
No response
The issue is on this part of the code: https://github.com/openstreetmap/openstreetmap-website/blob/master/app/controllers/api/notes_controller.rb#L250
This entire statement in the wiki is not quite correct:
Returns the existing notes matching either the initial note text or any of the comments. The notes will be ordered by the date of their last change, the most recent one will be first. If no query was specified, the latest notes are returned.
Search returns notes matching all of the specified search criteria such as text, user, date range. If no criteria are specified, it returns all notes. How they are ordered depends on sort and order parameters, with by last change most recent first being the default order*.
* also not quite correct as it seems, see later comments
If I query the notes "from" t1 (to now), I only get Note1. But I also want to retrieve Note2 because there was a comment after t1. The only way to get the Note2 from API is to put the "from" parameter as t2 before the creation of note2, but in this case, I will also get Note3.
No, you won't get Note3 if you use sort=updated_at. You can filter notes by time with either their first comment (when they are opened, sort=created_at) being inside the date range, or their last comment (their last update, sort=updated_at). Now if you want to find notes with any comment (not necessarily the first one or the last one) in a given time range, this is currently impossible. But it may not matter for you if you only specify the lower end of the date range (from) because the last comment is also going to be inside the range.
https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=5000
This returns one note, but if you add sort=updated_at, you'll get two:
https://api.openstreetmap.org/api/0.6/notes/search.xml?from=2021-01-01T00:00:00Z&bbox=-81.761460,3.912338,-81.416420,4.085293&closed=5000&sort=updated_at
sort=updated_at was supposed to be the default value, but https://github.com/openstreetmap/openstreetmap-website/commit/b84799f481e67c7a5cf6964e228c7a617b1defc7
Basically, always specify the sort parameter to avoid surprises.
The claim about sort=updated_at being the default value:
https://wiki.openstreetmap.org/w/index.php?title=API_v0.6&type=revision&diff=1959465&oldid=1954119
https://wiki.openstreetmap.org/w/index.php?title=API_v0.6&type=revision&diff=1959465&oldid=1954119
I reached out to Wiki editor, they replied with https://wiki.openstreetmap.org/w/index.php?title=User_talk:ENT8R&curid=192853&diff=2608297&oldid=2608258
Thanks for reaching out to me. I looked at the source code again and it looks like the side-effects of using the from parameter without explicitly specifying the sort and/or order are not documented in the Wiki right now. The issue is that in order to achieve backwards compatibility with previous states of the API (where sort and order did not exist yet), when using from to limit the results to a specific date range, this range is applied to the creation date of notes by default (necessary to keep old behavior) while the sorting happens based on the last update of the notes by default. This might lead to the confusion in the linked issue because it is not very clear which date (either creation date or last update) is used to apply the date range to when omitting the sort parameter.
