syncing with other tagsistant instances on other devices

[conrad-heimbold]

It would be cool if you could use tagsistant also on other devices (e.g. in an android app).

How can I sync a tagsistant database + files on one device with a tagsistant database + files on another device?

I think, this problem can be split into two:

1. How to deal with the files under ~/.tagsistant/archive or $MOUNTPOINT/archive ?
2. How to deal with the tags stored inside the database under ~/.tagsistant/tags.sql

Problem 2 shouldn't be too difficult, as SQL is already transactional and stores its history.

Problem 1 however could get more difficult.

EDIT: Removed idea for an Android App.

[StrumentiResistenti]

Seems like you're mixing two topics into one issue only.

The proposal of a sync procedure to exchange information between two Tagsistant repositories is an interesting point. I think that a migration tool or procedure would be really useful and I'm open to discuss this topic. Btw, I've even thought about a way to federate repositories, but it's somehow difficult to implement (duplicated inodes, spanning queries on multiple filesystems to be later reduced into a single set), so I would start this development effort only if motivated by users.

On the other hand, the idea of porting Tagsistant to an Android app sounds like turning an helicopter into a submarine. Tagsistant is a C application based on FUSE, a kernel infrastructure to write filesystem in userspace. Android apps are developed using Java (or another JVM language) and it's my understanding that the OS does not allow a user to mount a virtual filesystem. I don't even understand how you're suggesting to use the /proc filesystem. Moreover, an Android app (I've developed one) would reuse Tagsistant query resolution logic only (and perhaps autotagging and deduplication), but it should be forced to reimplement even that code portion in Java.

So, if you're proposing an Android app to browse (or even update) a Tagsistant compatible repository, I say I would find it useful. But I don't think there is a feasible way to port Tagsistant codebase into an Android app.

[conrad-heimbold]

The Android App could just save the "symlinks" to the files, and the tags of the files. The files themselves would just be stored on the sdcard. The app would just offer to browse the files, implementing the store/tags/archive/relations interface. When opening a file from this interface, the selected other app

• ether gets the direct file URL (file:///sdcard/... ) passed,
• or the tagsistant app moves the selected file into cache, and allows the other app to open/write the file inside its cache. When finished, the tagsistant app moves the changed file back to /sdcard/... .

EDIT: Removed some wrong stuff

[StrumentiResistenti]

I'm not sure to get your design clearly, but for sure no application is allowed to store anything in the /proc filesystem which is a virtual filesystem exposed by the kernel to get and set its configuration and to publish information about running processes and such. Anyway, I don't plan to develop an Android application, it's outside the scope of Tagsistant, but if you plan to develop it, I would be greatly interested because I own several Android devices.

Going back to the exchange procedure to migrate contents between repositories, I would say that a first check to do is if it's not already possible by mounting two repositories and cross-moving contents. To fully import a repository into another you could just:

1. scan the tags/ dir using find up to a depth of 3 and mkdir -p every entry in the destination repository
2. for each tag found in the previous step, copy all its content to the corresponding tag into the destination repository
3. scan the relations/ dir using find up to a depth of 3 and mkdir -p every entry in the destination repository

Of course point number 2 is highly unoptimized because each file with more than one tag would be copied and later deduplicated n - 1 times. Another approach could be to use the ALL/ dir and the .tags metafile. I don't guarantee this to be a bulletproof procedure, just a stub of what I would do on my own.

[StrumentiResistenti]

Try this procedure:

# mount source and destination repositories
tagsistant <src-repo> ~/src
tagsistant <dest-repo> ~/dest

# create a session tag to ease the migration
mkdir ~/dest/tags/merge-201702122134

# copy all the ALL/ tag content into the session tag
cp -vr ~/src/store/ALL/@@/* ~/dest/store/merge-201702122134/@@
[ ... cp progress here ... ]

# use the .tags metafile to move tagging
cd ~/src/store/ALL/@@/
for f in *; do cat $f.tags > ~/dest/store/merge-201702122134/@@/$f.tags; done

# delete the session tag
rmdir ~/dest/tags/merge-201702122134

# now just sync relations and aliases
cd ~/src/relations
find -maxdepth 7 -exec mkdir -p ~/dest/relations/{} \;

cd ~/src/aliases
cp -v * ~/dest/aliases 

[conrad-heimbold]

I would suggest to do (see step 4)

cat $f.tags >> ~/dest/store/merge-${NOW}/@@/$f.tags; done 

instead of

cat $f.tags > ~/dest/store/merge-${NOW}/@@/$f.tags; done

otherwise; you would just overwrite the tags; not sync!

[StrumentiResistenti]

I see your point, but doing an append will not solve this. *.tags files are parsed entirely. An effective way to preserve current tags while adding the merged ones is:

# use the .tags metafile to move tagging
cd ~/src/store/ALL/@@/
for f in *; do cat $f.tags ~/dest/store/merge-201702122134/@@/$f.tags | sort | uniq > ~/dest/store/merge-201702122134/@@/$f.tags; done

Not the cleanest command at all, nor the easiest to remember.

Btw, another problem of this naive procedure is that by listing objects inside ALL/ metatag, we force Tagsistant to add the inode in front of every name appearing more than once in the source repository. This means to me that a dedicated procedure is required to perform merging without hiccups.