/opt/cronicle/data/jobs - A lot of inodes, cleanup?

[vikingtoby]

Summary

/opt/cronicle/data/jobs is using a lot of inodes. Like, 318568, which is insane for the couple of MBs of data there.

Steps to reproduce the problem

We have some +300 schedules, with over 20k jobs completed each day.

Your Setup

Using https://github.com/bluet/docker-cronicle-docker in a containerized environment as statefull sets, only persistent folders are:

• /opt/cronicle/conf/config.json
• /opt/cronicle/data
• /opt/cronicle/logs

Cronicle software version?

Version 0.9.7

Are you using a multi-server setup, or just a single server?

Single master, three workers

Are you using the filesystem as back-end storage, or S3/Couchbase?

No, local

[jhuckaby]

This is normal. Cronicle doesn't use a database, so it writes many files to disk. Every job consists of many different files, for keeping track of things like the event history, combined history, job stats, job log, and possibly activity log if the job failed. Also, each of these lists has a "head" and many "pages" which are all files. Also, files are stored on disk like a cache, using a hash directory structure, many levels of directories deep. This is done for performance reasons, but it results in a large amount of INODEs used.

You can try setting your job data to expire earlier: https://github.com/jhuckaby/Cronicle#job_data_expire_days

[dheise-rheinenergie-trading-com]

We have a slighty similiar workload. For the same reason we set the job_data_expire_days to 30 days. You should also have a look on your file system backup and recovery times. The restore of the date directory could take a very very long time in case of emergancy.

[rommelandrea]

we have the same problem and even going to change the job_data_expire_days parameter but it seems that at the file cleaning level it is not respected correctly.

I will add additional information. After a crash due to inode saturation, we created a new disk and we move the storage to the new disk. We used the migration function to move the files and the result is that the application moved only the necessary files with a very important reduction of inodes (from 1.3M to 230K)

[vikingtoby]

After setting mine to 1 day, it seems to cleanup, but obviously, it's not a lot of history, but the setting certainly seems to work:

[tigerduck42]

Hi @jhuckaby,

is it safe to remove old file using a command like this?

This should remove all files oder than 90 days ...

find /opt/cronicle/data/jobs -type f  -mtime +90 -exec rm {} \;

Cheers Martin

[jhuckaby]

@tigerduck42 Yup, that's relatively safe. Those files are not part of the "database" index system, so you can go ahead and delete them.

[tigerduck42]

The only issue is, that on the "Event history" page of the events I still have entries, which spill an error because the detail information is no longer available.

How can I delete these entries as well? It looks like they are stored in the data/logs sub path, but don't get deleted by my command. Is it possible, that they have a newer modification time than the information displayed on the page when I click on a log history entry?

[jhuckaby]

There's no way to do this with v1, sorry 😞

v2 will have a way to batch-delete all jobs for an event.

[tigerduck42]

Perfect :-) Any estimate, when v2 will be available?

[jhuckaby]

@tigerduck42 Alas, I don't have a good estimate, but I'm shooting for late 2024.

Life keeps getting in the way 😊

[tigerduck42]

No worries, thank you for your support and product.