Reduce classifier job memory consumption

[kennylevinsen]

Describe the bug Having run photonix to import a library of ~11k images, I observe the following idle memory consumption:

type	resident memory	virtual memory
classification_location_processor	544 MiB	1489 MiB
classification_color_processor	408 MiB	943 MiB
thumbnail_processor	236 MiB	892 MiB

The reason that neither object nor style processors are on the list is due to consistently running their heads against the OOM killer. For reference, the VM is dedicated to photonix, and has 4GiB of RAM and 2GiB of swap, which I already found to be overkill by a factor of 4.

The contantly killed object and style processors mean that most images have no such information attached to them.

To Reproduce Steps to reproduce the behavior:

1. Setup photonix to a library of images on a machine with 4GiB of memory. I used mine with 11k images - not sure if amount matters
2. Let it run
3. Observe massive memory consumption and OOM killer in action

Expected behavior Negligible memory consumption on idle (e.g. <100MiB), normal memory consumption on operation (e.g. <2GiB)

Additional context Running off my other PR in order to be able to ingest my library in the first place.

[damianmoore]

Thanks for the useful feedback @kennylevinsen. This seems like a critical bug we need to fix before the 1.0 release. There hasn't been much testing on large photo libraries yet so we need to do more in that regard.

[spupuz]

has this been ficed i would like to test it for my 12k photo, actually testing photoprism that has a huge footprint on CPU

[damianmoore]

@spupuz It's memory usage should keep growing. Upon startup, it determines how many classification workers to start up based on the amount of RAM available. There has been some optimisation in this area but we want to do much more. Please let us know if you give it a try and don't give it your whole library at once.

[damianmoore]

Now that we have cron running inside container due to #124 (face recognition) I plan to utilise this to save memory. These classification processors are running all the time in the background, started by Supervisor. I'd like to have them triggered by cron each minute or so. Here is the proposal:

1. Classification processor triggered
2. If classifier is already running (detected via Redis lock) then exit
3. Otherwise, if there are Tasks waiting, grab lock and process a batch of them
4. If there are no more Tasks then release lock and exit
5. Otherwise continue with next batch

This should still be efficient as the model will be kept in memory until queue is depleted. There will be a start up time when new photos are added.

And extra refinement would be to check how many other classifiers are running and do not start up another type unless there is enough free RAM.