Synchronization support tomboy-ng <-> Nextcloud Notes

[monperrus]

trafficstars

Problem to solve: having a fully working setup with Linux desktop program / server side code / Android app for taking and synchronizing notes.

Tomdroid is dead, and one of the best Android app for self-hosted note taking today is Nextcloud + Nextcloud notes. So a potential solution is :

• Linux desktop program: tomboy-ng
• server side code: Nextcloud notes server with REST API
• Android app: Nextcloud notes app

Any hint on how to proceed would be appreciated.

[davidbannon]

Sigh, it has to be JSON doesn't it ? Hope you are better at JSON than me.

I'll have a read of that API but it looks like it might be a lot better documented than the Tomboy model was. Could be good ...

Davo

[monperrus]

Yes, this is JSON. I agree that the documentation of https://github.com/nextcloud/notes/blob/master/docs/api/v1.md is nice!

Looking forward to your thoughts.

[davidbannon]

OK, had a quick and very superficial read.

The nextcloud note attributes seem to map fairly well to tomboy, only exception is favourite, maybe thats "open on start" but I don't think so. So, we'd need to have some lookup table or modify the tomboy note format, both ugly. The note ID is not going to be a 1:1 transfer, tomboy uses GUID and nextcloud has just a int. So, again, some mapping. Maybe thats version 2.0 ? The Title and Timestamp looks easy.

The API looks like it it covers what we need, simple Gets and Puts.

The tomboy model assigns a Sync ID to make sure it is talking to the right repository, maybe we can overlay the username for that. You would assume a nextcloud would have only one note repository ?

I think I would take the same approach here that I took with Tomdroid, I would not consider Nextcloud as just another note in the sync model. Nextcloud does not, and cannot easily, keep the necessary metadata. So we need to keep it locally, per connection. We end up with one tomboy-ng node that syncs to Nextcloud in a one to one sync. That tomboy-ng node can of course also sync to other tomboy nodes using its normal sync model.

Just to be clear what I mean, so you can tell me if I am being silly -

Suppose I have a Linux laptop, a Linux desktop, occasionally reboot my laptop into Windows. And an old Mac I use to run some tests. All of these systems are part of the 'normal' tomboy file sync.

But on my Laptop I can also do a 1:1 sync to Nextcloud so that my Phone and Tablet can also see my notes.

I cannot do that 1:1 sync from any other system than the Laptop but because the Laptop auto syncs to my file repo and so do the other systems, its all in sync.

We would translate the Tomboy markup to md at sync, again, we'd loose some data but I think it would be acceptable.

What do you think of that model ?

I believe its pretty easy to implement.

Another approach might be to make a Nextcloud note that holds the meta data. As long as user did not mess with it, it would work, I think. Risky IMHO.

Davo

[davidbannon]

monperrus, have a had a better read and a think about the issues. A bit rambling but enough for you to see that there is issues we have to solve. One of the biggest problems with syncing is knowing which files have changed and which have not. Thats certainly the issue here.

If we download a list of notes on nextcloud, we get a noteID and a timestamp. Neither are particularly tomboy compatible, the noteID is an integer, the time stamp is a Unix one, so only one second resolution, Tomboy uses seven decimal places of a second. That's actually important, the chances of a note being edited at two places in exactly the same second is small, in the same 100 nS is negligible. But we will have to work to the second !

The Nextcloud noteID is an integer, presumably starting at 1 (or 0). So, if you have to merge two sets of notes, or move from one NextCloud to another, we cannot guarantee that the noteID is unique. Tomboy uses a GUID, unique (enough). Once a note has its GUID, Tomboy does not care about the actual format so we could, perhaps clear the last six digits and put the Nextcloud number in there, with some marker to show what we have done.

Usual Tomboy file name -

e8cb904d-2c4b-4ab7-8d93-3d353fcbdda0.note

A note derived from NextCloud, say, 177 -

e8cb904d-2c4b-4ab7-8d93-3d353-000177.note

That now allows us to recognise this note as one we should compare to the Nextcloud note number 177.

However, when I send a Tomboy created note up, say -

627e0bde-99ca-4377-9c73-92dcca942680.note

Next cloud will change its ID to, say, 178 and at the next sync, we have no way to tell we need to compare it to the local 627e0.... If the POST/Note API returned the newly assigned Nextcloud ID then we could maintain a map of the IDs used at either end. But we'd need a process to remove entries relating to deleted notes etc. Do-able but messy.

The Tomboy sync model stamps a SyncRevision number on every transaction and a list of all the current notes and their latest revision. That works if the "remote" is happy to maintain that data for us. I see no option to do that on NextCloud Notes.

In my Tomdroid sync, I rely a lot more on the LastChangeDate, I read the last change dates of all notes in Tomdroid and bring it back to tomboy-ng to compare. We can do that but only to one second resolution....

Deletions on NextCloud (assuming we solve the above) If we receive a list of notes from the Nextcloud server and we notice we have one locally that is no longer there on Nextcloud, we assume its been deleted up there and should be deleted here. Thats OK.

Deletions on Tomboy Notes deleted locally that have been previously synced have their IDs recorded until the next sync, then the NextCloud is told to delete them. There is an API for that purpose.

Need for ServerID The deletion models require a guarantee that we are talking to a system we have synced to before. Else we get a real mess on our hands. So, both the Tomboy and Tomdroid sync models stamp an ID when a new sync repo is established, it asks about that ID when making any subsequent connections, if they don't match, do not proceed ! That ID, or some sort of token, really needs to be stored at both ends !

Note Title is not useful in identifying notes, user's change the title all the time.

A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?

Sorry to be so picking, I am concentrating on the problems so, once we have solved them, its full steam straight ahead !

Note: if I was to redesign a new Tomboy schema, I would include a new GUID or maybe a revision number at every edit, that way I'd always be able to tell if the remote one was the same as the local.

[monperrus]

Thanks a lot for the comprehensive analysis.

The metadata problem is related to the file format.

If we store the note in Markdown on Nextcloud, we can use a YAML frontpage to store the metadata.

[davidbannon]

OK monperrus, that sounds hopeful !

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ? If so, it would all become quite easy !

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian but when thats sorted, I will build a nextcloud set up and have a play ....

Davo

[monperrus]

I am currently concentrating on establishing an Ubuntu PPA and possibly getting tomboy-ng into debian

Super nice!

but when thats sorted, I will build a nextcloud set up and have a play ....

Excellent, looking forward to it.

[monperrus]

Are you suggesting you could return a reproducable UUID (that you have allocated in Nextcloud) when the GET /NOTES call is made ?

There is already one, the id field:

[
    {
        "id": 76,
        "modified": 1376753464,
        "title": "New note",
        "category": "sub-directory",
        "content": "New note\n and something more",
        "favorite": false
    }, // etc
]

Am I missing something?

[davidbannon]

"id": 76,

The problem is that the ID is not unique. If someone syncs to their Nextcloud and later re-installs nextcloud or syncs to another install for some reason (and people do do that all the time !) they come across that same number again, because its got the same ID, its assumed its the same note. So, it overwrites it. Every Nextcloud Notes install will have a number 1, a number 2 and so on.

Truth is, Nextcloud Notes needs to assign a unique ID to every note even if it does not sync with Tomboy !

Davo

[monperrus]

Do you mean not unique per user? per server?

[davidbannon]

I mean it needs to be unique. Unique per server is not enough, suppose someone moves to another server that already has, perhaps one note. Unique per user is not enough, what happens when two users swap notes? Users send notes between them, there are, perhaps ten Tomboy format notes in the tomboy-ng install kit.

If all you want to do is 'import' or 'export' from one platform to another, then, its easy, no need to worry about unique, but if we are going to track them over some sync, edit, sync, delete cycles, we have to be sure we are talking about the same note.

Davo

[monperrus]

I understand. What about putting the unique id as a metadata?

[davidbannon]

Nine days ago I said

"A lot of this could be solved if we could find a way to store some metadata on Nextcloud. Could we, for example, just have another document there, created by the sync process ? ( Or, better still, is there any possibility we can update that document when a note is changed/deleted on Nextcloud ? I understand, a lot harder.) But just a nextcloud doc we can upload and download outside of the note system is, perhaps, what Nextcloud is all about ?"

Davo

[monperrus]

Hi Davo,

The solution I proposed is to store the note in Markdown on Nextcloud, and to use a YAML front matter to store the metadata.

[davidbannon]

You did indeed. And I thought that was a great idea. But you followed it up by saying -

There is already one, the id field: ... "id": 76,

Now, the way I understand it, everyone with more than 76 notes in their repository will have a note with ID = 76. So, IMHO, thats not unique. If you can replace the ID field with a UUID/GUID then it would be unique, that would be cool, we would be underway. But if the ID is allocated by ++(previous_ID) then we do not have a unique ID.

Just about every programing language these days has a library function to generate a UUID, its a reasonably clearly defined way to produce a (almost) guaranteed unique string.

I hope this is clear, we seem to be going around in circles here I am afraid.

Davo

[monperrus]

AFAIU, we can have two separate identifiers:

• the "physical" id of the REST api, used in the URLs, it is not unique
• the "logical" id of the note, generated on our own with a uuid library, stored in the front matter

Makes more sense?

[davidbannon]

That sounds great ! I assume you can include the "logical id" in the initial REST response, along with the physical ID ? That would work. I don't necessarily need to use the logical ID when manipulating notes, as long as I have seen a confirmation that the mapping between them has not changed.

I'll need to do some thinking about how we handle deleted notes. When I see a LID (Logical ID in your terms) in Nextcloud, that I don't have locally, its either a new one that I have not seen before (action=download) or one I have seen before but decided to delete locally (action=remote delete). With my Tomdroid sync hack, I keep a one to one sync (and a list of deleted notes) from one instance of tomboy-ng to one Tomdroid phone. I think that may be better than making the Nextcloud Notes one participant in a Tomboy file sync but it will need some work to be sure.

Anyway, if we have a transport model worked out, a sync policy can be worked out too !

Davo

[monperrus]

I assume you can include the "logical id" in the initial REST response, along with the physical ID ?

I'm not a Nextcloud developer, I don't know whether we would be successful in pushing this change.

Is that required for our use case?

[davidbannon]

Yes, I think its necessary. Maybe we might get away with a having a (nextcloud?) document that maps UUID to NCNoteID but it would be fragile. Lots of things that could break it that we may not think of.

For example, and a test you can do perhaps ? Make a new note, its gets assigned, say, number 44, can you see its number ? Delete it immediately and create a another one. Does it get ID 44 or ID 45 ? If it gets 44, no, we cannot work like that. If it gets 45, maybe. I'll have to do some work to map out the risks.....

Davo

[monperrus]

Does it get ID 44 or ID 45 ?

I don't know.

If it gets 44, no, we cannot work like that

I am not sure, thanks to uuid metadata

• Make a new note with uuid metadata xyz, it gets assigned, say, number 44
• Delete it immediately
• Create a another one with uuid metadata abc. Even if it has URL ID 44 again, we know that the uuid is abc, so it is a different note. The UUID metadata would be the reference not the server id.

[davidbannon]

Another long answer I am afraid. I don't think its hopeless but there are some serious downsides.

I don't think you can associate a note with a particular UUID if we cannot add it to the note's own meta data. Maybe I am wrong ? You understand what happens inside Nextcloud far better than I do so, perhaps you have a plan that works ? Can you explain it please ?

In my mind, if the UUID is not part of a note's metadata then we have to have some sort of lookup table. - ... ID=44 UUID=xyz ...

However, if the act of deleting that note and making a new one that uses the same ID does not automatically update the table, then we cannot tell it has happened, it looks like an edit. So, the process is perhaps -

• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We get the content that was in the first note to be called ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz
• We then can update, one way or another, the Tomboy note 'xyz'. Good.
• Next, the Nextcloud user does as you say, deletes the note and create a new one, also assigned ID=44
• *** critical *** Does something update the lookup table here ? How or where is the new UUID of 'abc' you mention assigned ?
• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz (but its a new note, user deleted one 44)
• tomboy-ng now overwrites the first ID=44 note with the new content and keeps its old UUID. For all intents and purposes, its the same note, it just has new content and a new title.

Maybe thats an acceptable departure from the Tomboy model. It means for example I cannot backup the old, deleted content as per Tomboy practice. If there is a sync clash, in tomboy-ng I show a diff of the two notes, thats going to be a messy diff to read where everything has changed. If someone does a restore from snapshot, again unexpected results because we are reusing the original UUID, 'xzy'. Note creation date will be the time when the very first ID=44 was seen by tomboy-ng, that ID may well be reused many, many times with new notes but each one will still show the quite incorrect note creation date of the very first note to be assigned that ID.

I suppose we could take the view that a sync with NCNotes is not a normal Tomboy sync, as long as the end user understands there are times when unexpected results will occur.

I understand you don't think there is any prospect of getting the NCNotes people to add a UUID and/or a creation date ?

Davo

[monperrus]

Hi Davo,

Here is my understanding of the scenario (in bold my explanations)

• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• We get the content that was in the first note to be called ID=44.
• We lookup our table, find ID=44 in there and its associated with UUID=xyz
• We then can update, one way or another, the Tomboy note 'xyz'. Good.
• Next, the Nextcloud user deletes the note and create a new one, also assigned ID=44
  • If the user creates the note on Nextcloud directly, there is no UUID. Because there is no UUID, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44
  • If the user creates the new note on tomboy-ng, tomboy-ng knows that how to update its internal lookup table for ID=44
• We trigger a sync between tomboy-ng and Nextcloud Notes
• We use the NCNote api call, get a list of notes including one with ID=44.
• Because there is no UUID in the metadata of this note, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44 with the newly created uuid

--Martin

[davidbannon]

Right, I think we are focusing in here.

Next, the Nextcloud user deletes the note and create a new one, also assigned ID=44

If the user creates the note on Nextcloud directly, there is no UUID. Because there is no UUID, tomboy-ng understands that 1) a new UUID must created 2) it must update its internal lookup table for ID=44

No, there is still an entry in the table for ID=44 : xyz, no one has deleted it. tomboy will not not be able to detect that the note has been deleted and reissued. So, will use the old UUID. (different result if user sync after every delete).

If the user creates the new note on tomboy-ng, tomboy-ng knows that how to update its internal lookup table for ID=44.

Yes, in that case, tomboy-ng will know to remove to old entry for 44 from the table. It will request a new note on NextCloud (hmm, does NextCloud tell us the ID ? Thats probably another problem, if it does not get told the ID of the new note, it cannot add an entry to the table.).

I am starting to think an export / import filter is practical but not real syncing. No prospect of interesting the NextCloud developers in adding a UUID to a note's metadata ? It would be useful for a lot more applications than tomboy-ng.

Sorry I am being so negative !

Davo

[davidbannon]

OK monperrus, I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, I have run up some six notes and each one was (substantially) greater than the previous one. The sequence was -

136 155 (deleted) 193 212 (shutdown and restart) 339 Maybe its elapsed time based ?

So, if they never go back, that does make them kind of unique, doesn't it ?

Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

I would like a statement from the developers that the number will always keep growing, do you want to ask them or should I ?

Davo

[monperrus]

I have built a nextcloud install (wow, finally found a use for Snaps!) and have some more data. Interesting.

cool :)

Note IDs are not allocated sequentially, indeed they jump a lot more than one per note, Not unique across platforms but I guess if we warn users about that issue they can be alert to danger.

Agree, this is good enough IMHO

I would like a statement from the developers that the number will always keep growing

I've browsed the Nextcloud code.

AFAIU, the id comes from method getId from interface Node with the implementation of FileInfo.

This id is actually a database field called file_id, confirmed by https://github.com/nextcloud/notes/blob/017e825dafd4212c362197f7def97bf225f17997/lib/Db/MetaMapper.php#L42

It seems that this database field is created by $fileId = $builder->getLastInsertId(); see https://github.com/nextcloud/server/blob/67ecdc969cd93442918180e179e1f05ed8618d92/lib/private/Files/Cache/Cache.php#L295

Per my understanding of the database code, this is is unique and most likely increases monotonically.

Good news!

[davidbannon]

Thought you might like an update. I have not dropped the ball.... I have working now a rough mockup of a tomboy-ng to NextCloud sync. While its working and I think, reasonably satisfactory, it does have some limitations that end users will need to work with. For the record -

• I have to be sure that I am talking to the same server each time we sync. (Otherwise, I see a missing note at the far end and delete it locally for example!) So, during the initial 'Join' process, I make a new note that contains a UUID. If the user renames that note, deletes it or alters the UUID line, bad things happen ! I'll put some clear advice in that same note.

• Because NextCloud does not assign a unique ID to a note, I have to keep a mapping of UUID to the current NextCloud assigned ID. That mapping becomes invalid if you move your notes to a new NextCloud provider or rebuild your own install for example. So, if you then disregard the advice you get and proceed to make a new 'Join' you get duplicates of all notes that exist at both ends. This problem can be managed by syncing before abandoning the old NextCloud server and ensuring there are no notes manually transferred from it to the new server. You allow tomboy-ng to populate NextCloud ....

• NextCloud does not support some of the Markup used in the Tomboy standard. So, if you use underline or highlight, or larger fonts mid paragraph, that markup (but not the text) will be lost. Small font is similar but as there is a well supported and reasonably unobtrusive markup for that (that Nextcloud does not render) I have left that in, if you don't alter it, it remains through a sync cycle. If you don't like it there, don't use it. _{Github supports it}

I have not yet addressed any SSL issues. Nor keeping the NextCloud password in a safe place, thats looking like a platform by platform problem, sigh ....

I am also a bit concerned by the extra bulk this is adding to tomboy-ng, there is quite a lot of extra code required, particularly for the TomboyXML to markup conversion. I really don't know how many users will find this a useful addition. One approach may be to make a separate, standalone app that does the sync, only the tomboy-ng AND NextCloud users would need it. It would have to talk to tomboy-ng when sync-ing, else we'd risk changing a note just as it was synced.

Hmm....

Davo

[monperrus]

I have working now a rough mockup of a tomboy-ng to NextCloud sync

Oh great! I volunteer to beta-test it!

I am also a bit concerned by the extra bulk this is adding to tomboy-ng,

I think it's a great idea to keep it here, in order to have easy deployment and dissemination in distros having tomboy-ng, esp. Debian

[davidbannon]

OK, I have a working system I think. However, I have not tested it against an SSL Nextcloud server. Seems I cannot get SSL onto my Multipass Nextcloud (because it does not have a proper FQHN and the commercial provider I signed up for a free account with, QLoad (chosen at random), does not have the notes application installed and does not let me install a new application of my choosing. Is that normal ?

So, can you recommend a more suited provider ? I'm reluctant to go for a paid one, have no need for it. Or do you have a system available to try out against ? If so, let me know what OS you want the tomboy-ng client to run on (and assure me you understand about backups).

David

[sliwowitz]

Or do you have a system available to try out against ?

I have my own SSL server, I can set up a nextcloud account there for you to test if you want.

[davidbannon]

Sure, that would be great, thank you ! I would need only Notes, a few hundred megs, no more than a gig and that only for a short time. User=dbannon if possible. Davo