Task history

[Enet4]

trafficstars

The current Dicoogle task list has a few issues:

• Information about all tasks is kept in memory, even after they were terminated, and until they are deleted.
• In order to reduce clutter, they must be deleted manually with the Close operation (either through the UI or the API). Once that is done, information about this task needs to be searched in the main log, which is inefficient.

The proposal sums up to the following:

• There would be a persistent registry (log) of task execution and termination, containing at least timestamps, UIDs and statistics of terminated tasks.
• Tasks are never closed. Instead, they are archived by flushing information about them into the registry, freeing them from main memory and hiding them from the main list.
• The core Dicoogle would include a simple task garbage collector which archives old terminated tasks once in a big while (scheduling and task time-to-live should be configurable).
• The user interface would normally only show recent (non-archived) tasks. A separate interface could be used to retrieve a history of old tasks on demand.

[nuno-loucao]

Hi,

Is there any way to setup dicoogle in such a way that it flushes all completed tasks automatically?

Thanks, Nuno

[bastiao]

I think if is run with -Ddicoogle.tasks.softRemoveTasks=1, it should flush empty index queue history.

[Enet4]

Thank you for reporting. It was not yet mentioned here, but as @bastiao mentions we added a task cleaning procedure in #441 (it entered in Dicoogle 2.5.2) to mitigate the problems reported in this issue. It is configured through JVM parameters as follows.

• -Ddicoogle.tasks.softRemoveTasks=50000: if the number of completed or cancelled tasks reaches this threshold when a new task was added, Dicoogle will automatically clear a batch of old tasks. You can reduce this threshold to 1 or 0 to keep the number of visible tasks to the minimum.
• -Ddicoogle.tasks.numberTaskClean=2000: the number of completed or cancelled tasks to clean in batch
• -Ddicoogle.tasks.removedCompleted=true: enable/disable this cleaning process, enabled by default

[nuno-loucao]

Hi,

Thanks for the quick reply. I've launched dicoogle with the following command (i'm using macos): java -jar dicoogle.jar -s -Ddicoogle.tasks.removedCompleted=true -Ddicoogle.tasks.softRemoveTasks=3 It didn't work. Only if I restart the app, not only logging out, does the system clear the completed tasks.

Also, i can open a new issue, the indexing is not working. I've tried to unindex the files from the database, but it doesn't work (neither unindex nor remove physically).

Thanks for your help. Best, Nuno

On Mon, Apr 15, 2024 at 5:38 PM Eduardo Pinho @.***> wrote:

Thank you for reporting. It was not yet mentioned here, but as @bastiao https://github.com/bastiao mentions we added a task cleaning procedure in #441 https://github.com/bioinformatics-ua/dicoogle/pull/441 (it entered in Dicoogle 2.5.2) to mitigate the problems reported in this issue. It is configured through JVM parameters as follows.

• -Ddicoogle.tasks.softRemoveTasks=50000: if the number of completed or cancelled tasks reaches this threshold when a new task was added, Dicoogle will automatically clear a batch of old tasks. You can reduce this threshold to 1 or 0 to keep the number of visible tasks to the minimum.
• -Ddicoogle.tasks.numberTaskClean=2000: the number of completed or cancelled tasks to clean in batch
• -Ddicoogle.tasks.removedCompleted=true: enable/disable this cleaning process, enabled by default

— Reply to this email directly, view it on GitHub https://github.com/bioinformatics-ua/dicoogle/issues/302#issuecomment-2057310323, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHTMQKJ2J72PZXKWWNJ4OUTY5P66RAVCNFSM4D64RPC2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBVG4ZTCMBTGIZQ . You are receiving this because you commented.Message ID: @.***>

-- Nuno Loução Research Scientist @ Digital Surgery Lab Champalimaud Clinical Centre Champalimaud Foundation +351930952230