Configuration Option for Time to Keep Logs

[mircealungu]

The Zeeguu deployment of the FMD is about one year old and it got quite slow. I wonder what's the best way to handle the situation when FMD DB grows to the point where the dashboarding loading is too slow.

To do:

• look at what slows down the system for big databases; if it's only some missing indices that could be solved easily

To think about:

• allowing fine-grained configuration (e.g. throw away the profiler info after one month but keep the request duration for six months and the number of hits for ever?)
• allowing some kind of data warehousing where the data is dumped onto a secondary storage

[bogdanp05]

Hi @mircealungu! I think this is a cool problem that we thought about during my thesis, but eventually scoped it out. My input:

• Indices: right now we don't define any indices in SQLAlchemy, so the only ones created are for primary keys and I think MySQL creates them for foreign keys too, but I'm not sure. You can test this approach by manually creating some indices in your db and see if performance improves. Then we can updated our model definitions accordingly.
• Data removal: rather than automatic periodical deletion of old data, I would opt for a simple cli utility that you could run whenever you want and it would be something like: fmd_trim --profiler -t 3m which would remove all profiler-related data older than 3 months. And this solution could be integrated with a data warehouse.

[safeRetail]

Hi @mircealungu

I am facing same issue. The dashboard get slower than before. A quick check on the size of the sqlDb show me that the db is up to + 2Gb ....

It will be a good feature to have the possibility to cleanup database.

Any suggestions ?

Thanks

[mircealungu]

Hi @safeRetail ! Thanks for joining the conversation!

At the moment we do not have the manpower to implement this feature :( Simplest thing is to delete the DB at the moment.