superset icon indicating copy to clipboard operation
superset copied to clipboard

feat(python): add purge Event logger database

Open jplanckeel opened this issue 2 years ago • 2 comments

SUMMARY

An example to add purge for Event Logger in Database, because today Logs tables is not purged and grow to infinity.

Is not perfect, ! :warning: warning this PR can slow down Superset Interface if you have a lot of element in logs table.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

image

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • [ ] Has associated issue:
  • [ ] Required feature flags:
  • [ ] Changes UI
  • [ ] Includes DB Migration (follow approval process in SIP-59)
    • [ ] Migration is atomic, supports rollback & is backwards-compatible
    • [ ] Confirm DB migration upgrade and downgrade tested
    • [ ] Runtime estimates and downtime expectations provided
  • [ ] Introduces new feature or API
  • [ ] Removes existing feature or API

jplanckeel avatar Jul 07 '22 09:07 jplanckeel

Hi everyone ! Do you think this PR could be merged as is ?

After test we need an index on dttm column for table logs ;)

jplanckeel avatar Sep 15 '22 13:09 jplanckeel

Ping @eschutho @geido please (i don't know if you are the good persons to ping, but we found nobody in the CODEOWNERS :'-(). Do you think this kind of purge could be a good idea ?

If not, we will manage it in our own superset config code.

:pray:

Adding some people who might have a better understanding of this part of the application

geido avatar Oct 03 '22 12:10 geido

Ping @jplanckeel, do you think these changes could be done on our way ?

i will test to copy prune report.log ;) https://github.com/apache/superset/blob/master/superset/tasks/scheduler.py#L99

jplanckeel avatar Oct 19 '22 09:10 jplanckeel

i create a new PR with the new solution to prune logs table with celery task https://github.com/apache/superset/pull/21873/files

thank for your reponse ;)

jplanckeel avatar Oct 19 '22 13:10 jplanckeel

@jplanckeel closing this in favour of the new PR

dpgaspar avatar Dec 14 '22 22:12 dpgaspar