Adds note versions and variable note tags

[nenad-vujicic]

Description

PR proposes adding support for note versions and variable note tags. Here is a detailed list of changes:

1. Updated and created database tables notes and note_versions for keeping note versions like it is done for other elements (nodes, ways and relations)
2. Added creating new version whenever note is reopened / closed / hidden and improved creating / saving notes by using same methods (from_, save_ methods) like it is done for other elements (nodes, ways, relations). Also, ID of generated note-comment is kept in note_versions for easier reconstructing note's provenance.
3. Added generating new versions from note comments (for every reopening, closing, hiding new note version is generated and ID of appropriate note-comment is kept for easier reconstructing note's provenance)
4. Created database tables note_tags and note_tag_versions for keeping versioned note tags
5. Added support for creating and saving versioned note tags (from URL encoded parameters)
6. Added support for updating notes properties (description, latitude, longitude and tags). To update a note, one should send PUT HTTP request with FULL note definition from which new note version content will be reconstructed (similarly when creating)
7. Added basic testing of above functionalities

I have already opened first PR which does the first step, but wanted to show how it can look like at the end and would like to read your comments about it.

Please, let me know what you think about this.

Thanks.

How has this been tested?

Automated tests and manual testing by injecting code snippets after creating notes datasets

Here is how it should look like

Displayed resolved note:

Displayed opened note:

XML output for "opened note":

If we decide to display note's status as a note-tag, it can be done by changing 4-5 lines of code:

[github-actions[bot]]

	1 Warning
:warning:	Number of updated lines of code is too large to be in one PR. Perhaps it should be separated into two or more?

Generated by :no_entry_sign: Danger

[nenad-vujicic]

Updated PR to display how notes versioning, removing special note comments (ones with non-present body which are not "commented") and versioned note tags could look like.

Removal of special note-comments (while keeping backward compatibility, i.e. returning same "discussion" with hopefully similar performances) with non-present body is implemented by:

• creating a view which "merges" note_versions and note_comments. I used this approach to keep note comments preloading etc.,
• switching .comments and .all_comments associations to point to newly created view, instead of NoteComment,
• removing adding special comment if body is not present
• running migration script which deletes all special note comments with non-present body.
• After this, we should be able to remove event column (still not implemented).

I would highly appreciate any comment about these changes.

[nenad-vujicic]

Updated PR to display how:

• notes versioning,
• removing special note comments (ones with non-present body which are not "commented"),
• removing event column from note_comments ,
• versioned variable note tags (creating, reading, updating, deleting),
• retrieving specific note version (API only),
• retrieving note history (API only) and
• note redactions (API only)

could look like.

[simonpoole]

1. for keeping note versions like it is done for other elements (nodes, ways and relations)

I continue to think that this is not a good idea, for starters notes are not like or even the same class of entity as OSM elements.

Then in the current model, if you so want, notes are ledger-like (think block-chain), you are adding entries to a sequential list with immutable entries, versions literally don't fit in to such a model, nor is there even any kind of indication why we would need versions and what they would be useful for.

[nenad-vujicic]

1. for keeping note versions like it is done for other elements (nodes, ways and relations)

I continue to think that this is not a good idea, for starters notes are not like or even the same class of entity as OSM elements.

Then in the current model, if you so want, notes are ledger-like (think block-chain), you are adding entries to a sequential list with immutable entries, versions literally don't fit in to such a model, nor is there even any kind of indication why we would need versions and what they would be useful for.

@simonpoole thanks for feedback! Just to clarify your idea, is #5344 plus update functionality (we would have new event type - update) something you would like? I believe we could add to "(special) update comment" changes we made (description, lat / lon, tags) if we want to have note's provenance and it will not make Discussion thread too dirty (in that case Discussion will be more History than discussion :worried:).

[mnalis]

I think this PR would benefit from outlying in more details exactly why is it doing changes (i.e. what exactly new user-requested functionality will become available), instead of just enumerating low-level technical steps which are being done. Judging by few words and linked PR discussion, I guess it might have something to do with enabling users to add tags to notes and thus making notes mutable, but I cannot really tell what is an actual idea being implemented, and what are technical side-effects.

plus update functionality (we would have new event type - update) something you would like?

I don't think that notes would benefit from being changed, in fact it would likely be very bad. I.e. if note could be moved or its description changed, followup comments could be misunderstood or turned out of context. Also, it would make necessary not only API/UI to see what changes were being made and when, but a new (or extended) planet.osm.org dump (like we have e.g. planet-*.bz2 as well as history-*.bz2 for nodes/ways/relations).

Only part of Note that should be modifiable IMHO is implementing #hashtags, the rest of the notes should better remain immutable. Having versions, locations and descriptions being changed are IMHO likely to introduce much more chaos then help.

TL;DR: can you list example use-cases and how this PR addresses them (i.e. "users currently do this thing xxxxx which is inconvenient, and with those changes implemented they will be able do this other thing which is easier/better because yyyy")

[nenad-vujicic]

I think this PR would benefit from outlying in more details exactly why is it doing changes (i.e. what exactly new user-requested functionality will become available), instead of just enumerating low-level technical steps which are being done. Judging by few words and linked PR discussion, I guess it might have something to do with enabling users to add tags to notes and thus making notes mutable, but I cannot really tell what is an actual idea being implemented, and what are technical side-effects.

@mnalis thanks for feedback! PR's main contributions are:

1. Adds mutable note tags. Why do we (both developers and mappers) need them? Do we want to continue writing hash tags (#surveyme and similar) instead of clicking "add tag" button (still not implemented)? Do we want to read hash tags from discussion's comments instead of having e.g. a table with tags and using appropriate API calls which will do this in an efficient and proper way (implemented)? Do we want to manually filter / search notes or use 3rd party tools across the www instead of having much better stuffs on one place (still not implemented)? I think the answer is "Yes, we would like to have add tag button, see a table with tags instead of #tag-name and searching / filtering ability on OSM". Why mutable? Some note tags are naturally static (e.g. created-by) while some are naturally dynamic (e.g. edited-by, language, ..), also we need to allow adding / deleting after creation (because of input errors, changing circumstances, ..).

2. Adds note versioning. Why do we (both developers and mappers) need it? I believe having an efficient way to retrieve note's provenance (mostly implemented) is important to enable building efficient notes history viewers like we currently have for elements (e.g. this) to enable moderators more efficient note's inspection (at the moment this is very slow and fully manual, but in future it will be much faster (still manual) at the beginning, semi-automatic at one moment and eventually almost fully automatic). Also, on OSM website's source-code level, we are moving "special comments" from note's comments to "version's parameters" which will bring us much better decomposition and easier maintenance (also, as a side-effect, we are preparing for removing special comments from Discussion, leaving it to only end-user's comments, in the next API increase).

plus update functionality (we would have new event type - update) something you would like?

I don't think that notes would benefit from being changed, in fact it would likely be very bad. I.e. if note could be moved or its description changed, followup comments could be misunderstood or turned out of context. Also, it would make necessary not only API/UI to see what changes were being made and when, but a new (or extended) planet.osm.org dump (like we have e.g. planet-*.bz2 as well as history-*.bz2 for nodes/ways/relations).

Only part of Note that should be modifiable IMHO is implementing #hashtags, the rest of the notes should better remain immutable. Having versions, locations and descriptions being changed are IMHO likely to introduce much more chaos then help.

At the moment, this PR supports modifying note's description, latitude / longitude and note-tags. At the beginning I wanted to support only modifying tags, but added support for other fields because of this. For me, it's not a problem to remove modifying note's description and latitude / longitude (in database table note_versions we would remove columns latitude, longitude, tile and description) and enable modifying only note-tags.

TL;DR: can you list example use-cases and how this PR addresses them (i.e. "users currently do this thing xxxxx which is inconvenient, and with those changes implemented they will be able do this other thing which is easier/better because yyyy")

The purpose of this PR is to show current proposal of how note's could look like. I will update this PR with new features until above functionalities are added and remove stuffs from it when they are committed / rejected. Use-cases we are trying to improve are (among others):

1. At the moment mappers use hash-tags. In near future we could use buttons for creating / updating / deleting note tags. Similarly for reading, instead of passing through note-comments, we will have a table like for elements (this is implemented).
2. At the moment mappers manually search / filter notes. In near future we could use UI for defining searching / filtering criterias.
3. At the moment mappers manually inspect notes or use 3rd party tools. We already have implemented (in this PR) retrieving particular note's version, which gives us a chance to build note's history viewers / analyzers for automatizing parts of note's inspection.

[mnalis]

TL;DR: great for adding support for adding/changing (optional) tags on notes, as well as (optional) extra API params / website changes to filter by them; but please no mutable note texts/coordinates - it would be quite problematic!

---

Thanks for clarifications @nenad-vujicic !

• At the moment mappers use hash-tags. In near future we could use buttons for creating / updating / deleting note tags. Similarly for reading, instead of passing through note-comments, we will have a table like for elements (this is implemented).

I agree; adding tags to notes is useful feature. Data consumers which do not implement that extension would still be able to use all existing functionality in the same way, so would only miss on those extensions.

• At the moment mappers manually search / filter notes. In near future we could use UI for defining searching / filtering criterias.

Having an API extension being able to also filter by those tags would be nice addition, getting around limitation of maxnotes and local filtering etc. (provided it reduces the server load instead of increasing it). Also, data consumers which do not implement those search extensions would retain same functionality as before, so this sounds good to me too.

• At the moment mappers manually inspect notes or use 3rd party tools. We already have implemented (in this PR) retrieving particular note's version, which gives us a chance to build note's history viewers / analyzers for automatizing parts of note's inspection.

Mutable text/coordinates of Notes however looks highly problematic to me.

Notes are currently journal-like implementation, i.e. only new comments can be added, and old ones cannot be changed. I think making the comments and coordinates mutable would be very bad. It is IMHO not comparable to elements (nodes/ways/relations) versioning, as those are self-contained and do not have dependent subelements. But notes have: each note comment is dependent "subelement", i.e. meaning of each note "reply" depends on the content of coordinates and original texts of ALL preceding comments, and not on their latest version!

Changing the text or coordinates of an original note (or previous comment) at a later time, (potentially) changes the meaning of all comments that were written between time where original text was written and its updated version. Also, some apps cache notes offline, which would further make determining on which version the reply was made problematic. Also, one should consider that there will be both old API apps and new API apps running at the same time for forseeable future, so they would need to work together without stepping on each other toes.

I.e. in current Notes system it is extremely easy and efficient to follow the temporal ordering of events, just follow the "comments" by increasing ids/timestamps. Any clarifications or corrections are just another comment down the line.

Making the comments text or coordinates mutable would totally break the workflow of all existing note consumers, and require all consumerts to convert to new API and also invest significant effort to make sense of and avoid misinterpreting Notes, for no good reason.

Also, how would the Notes planet dump change with mutable notes? It seems like a drastic change, perhaps even necessitating splitting to separate planet-notes and history-notes as we have for elements (compared to e.g. mere adding of note tags, which would probably just add "tags=" attribute to appropriate XML element, which would work as-is in most if not all data consumers, and is trivial change)

[nenad-vujicic]

Mutable text/coordinates of Notes however looks highly problematic to me.

...

Changing the text or coordinates of an original note (or previous comment) at a later time, (potentially) changes the meaning of all comments that were written between time where original text was written and its updated version. Also, some apps cache notes offline, which would further make determining on which version the reply was made problematic.

Yes, I agree with you. I'll probably (if others agree and they probably will :shrug:) leave only updating note tags in PR (and status, of course, through closing / reopening / hiding). Btw, I haven't planned to enable editing comments :satisfied:

Also, one should consider that there will be both old API apps and new API apps running at the same time for forseeable future, so they would need to work together without stepping on each other toes.

I haven't planned to change API, but only to extend it.

Also, how would the Notes planet dump change with mutable notes? It seems like a drastic change, perhaps even necessitating splitting to separate planet-notes and history-notes as we have for elements (compared to e.g. mere adding of note tags, which would probably just add "tags=" attribute to appropriate XML element, which would work as-is in most if not all data consumers, and is trivial change)

I think this is very good question even if we allow changing only note-tags, thanks!