Support many work directories

[jflesch]

It could be handy to have support of multiple work directories.

2 problems:

• Keep track of the indexes for each work directory (without losing/wasting disk space when one work directory is deleted)
• GUI

[tYYGH]

It could indeed be the first step to a whole new set of features based on “layers”/shares. Eg.

• just like in ownCloud each user could have their own papers rw, and also see papers shared by other users, either ro, or rw;
• or there could be a layer of documents shared by all users, and another layer of personal documents…

[pacien]

Maybe should the index be stored within each work directory instead of ~/.local/share/paperwork as well. I currently switch between multiple work directories by using a symlink-based workaround for both the work directory itself and the index.

[jflesch]

No, they won't be stored in the work directory:

1. Work directory must be backup-able easily. There is no reason to include the index in the backups
2. Work directory must be sync-able using tools like OwnCloud, SparkleShare, etc. You don't want to sync extra data for no reason (and worst, you don't want to keep the change history of this extra data)

I use another set of documents than mine, because otherwise SparkleShare would sync my tests so I often have the problem when developing on Paperwork. Therefore I gave it already some thoughts.

What I'm considering doing:

• No change at all in the UI.
• Instead of having the index and the label guessing data in ~/.local/share/paperwork/, it could be moved in ~/.local/share/paperwork/[hash of the work directory path]/

Then, the problem is just to delete automatically unused indexes. So I'm thinking the following rules can be applied:

• When Paperwork starts, it looks at every index in ~/.local/share/paperwork/
• If an index matches a non-existing path and if this index hasn't been used for more than 2 weeks --> deleted.

Indexes related to paths that don't exist anymore cannot simply be deleted immediately, because some people may want to store some of their documents on removable devices (USB keys, etc).

[jflesch]

Regarding the GUI, what I may be considering is just adding a simple menu "recently used work directory" somewhere. But quite frankly I'm still not clear on that part.

[pacien]

Good point regarding the separation of the index and the data.

However I would really be against a lookup+timer approach for the index deletion: I store collections on my NAS and I mount them only when needed, with every use arbitrarily time-spaced.

I would be in favor of a startup screen listing all (recently or not) used paths, giving the possibility to remove them from the list and delete associated indices. This would also avoid waiting for the integrity check of the last used work directory to complete to simply switch to another one. Some inspiration may be found in the IntelliJ IDEA startup window.

[jflesch]

However I would really be against a lookup+timer approach for the index deletion: I store collections on my NAS and I mount them only when needed, with every use arbitrarily time-spaced.

Worst case scenario, the index is deleted when it shouldn't. It will automatically be rebuilt when you use again that work directory. Since you wouldn't have used it for more than 2 weeks, I guess a 5~10min rebuild wouldn't be a big deal ;)

I would be in favor of a startup screen listing all (recently or not) used paths, giving the possibility to remove them from the list and delete associated indices

Hm interresting idea. However I'm not fond of having more screens/dialogs/windows. With too many of them, we will end up with a UI like the space shuttle cockpit.

This would also avoid waiting for the integrity check of the last used work directory to complete to simply switch to another one

I think you misunderstood what I suggested: For each index, the check will simply be "do the root of the work directory of this index do still exist, and if not, when was the last time this index was used ?". Those are really quick to check. Update: nevermind, I misunderstood you ...

Some inspiration may be found in the IntelliJ IDEA startup window.

IDE GUIs are exactly the way I never want to go. I'm trying to make a GUI that my own mother could use (and believe me, it's not a easy goal). From an ergonomic and usability standpoint, IDE are purely awful (which is not a surprise since they do try to please absolutely everybody). (one of the only exceptions I've found up-to-now is gnome-builder, but there is still some work before making it actually really usable ...)

[jflesch]

Having a popup when starting could work, but if we follow IntelliJ IDEA, it basically means you assume that everybody uses multiple work directories. By default, it should be assumed that user will use only one work directory (which, for instance, would be true for me if I were not Paperwork's main developer).

Other issue: It means the user will have to manage a list of work directories. It means extra work for them. Even if minimal, the goal of Paperwork is to make everyone's live easier and simpler, not more complex.

[pacien]

The rebuild time for a >500 docs collection is really a big deal when all you want to do is insert a new document or simply search for "that invoice" with someone over the phone waiting. I am really not fond of arbitrary timer-based approach. I guess that it should be at least be optional or configurable.

If there is no intermediate window allowing the user to load the collection of their choice, I assume that the last used directory will be opened automatically, triggering the integrity check I was referring to. I do not know if it is safe to load another directory while one is being loaded and its index possibly updated.

Agreed on the fact that UI should be simple. I think that a window listing used directories (like this one) would actually be simpler for the user, but this is just my humble opinion.

If none of these are going to be implemented, would it be possible (if it is not already the case) to pass a work directory and an index directory as command line arguments? I think that it would help some "advanced" users a lot while not adding clutter to others.

[jflesch]

I assume that the last used directory will be opened automatically, triggering the integrity check I was referring to

Yep, that's what I understood .. after writing my first reply :-)

I do not know if it is safe to load another directory while one is being loaded and its index possibly updated.

1. Index loading can be safely interrupted. There is only one exception: When it reaches the step "Rewriting index ...". This specific operation is atomic.
2. It is safe to request loading another directory while another is being loaded / updated. At best Paperwork will interrupt the previous loading/update. At worst, it will wait for it to finish and then only load the new work directory.

If none of these are going to be implemented, would it be possible (if it is not already the case) to pass a work directory and an index directory as command line arguments? I think that it would help some "advanced" users a lot while not adding clutter to others.

Sure, no problem.

[JoshuaPK]

I agree with @pacien . @jflesch , I didn't realize you had a use case where documents would be backed up to an rsync-like service such as OwnCloud, and you would not want to have to re-upload the index every time. For that use case it makes complete sense to store the indexes in a centralized or standardized location. Having said that, my use case doesn't involve a service like that. I want a directory containing everything that I can easily backup, burn to a DVD, expose via NFS, or whatever else. And I do not want to have to sit and wait for things to reindex on a periodic basis... whenever I add a new document, or when I ask it to reindex everything, at that time it should modify the index... but no time else.

Fortunately these two use cases are not mutually exclusive. :) We can make a config option for Index location. We could have it default to the standardized location that it is now, but it could be overridden by a setting in the config file, or on the command line. What do you think?

[jflesch]

@JoshuaPK : Backuping both ~/papers and ~/.local/share/paperwork wouldn't solve your problem ? oO

[jflesch]

I'm not against a new option in ~/.config/paperwork.conf. But it won't be visible in the settings dialog. You will have to edit ~/.config/paperwork.conf manually.