RTIInternational
WIP: Database daily ingest
This MR adds the daily ingest option to the External database in the project creation and update forms, and information for how to add this as a cron job.
@peridoteagle I've marked this MR as WIP since I haven't been able to test the cron command on the server. I like the way the DB dump emails the logs, and I think that sounds like a great idea if we can have this do that as well.
Pulled this branch to our staging site on http://cds-mallard.rtp.rti.org:8088/. Branch is still active if you want to try to replicate. There is a Database Connection project in root. Here's what I did:
-
Pulled branch and migrated DB
-
Attempted to go to Details -> Update Project -> Update/Remove Database Connection
-
Filled in test database credentials and selected box for Scheduled Ingest
-
Got these errors
/180490659-0b337b49-cb4c-4ad7-bfb7-18dbe0e7c521.png"> -
Confirmed that I can see the column in the DB
@peridoteagle looks like there were some database conflicts between this branch and npsas main. Not sure if that was the issue, but I pulled in the branch and resolved them. Would you mind pulling, migrating, and trying again?
@peridoteagle This should be ready for you to test now, it's up to date with dev and migrations have been merged
Summary of my process here: Had some issues with the cron jobs not working. Alex H suggested using schedule. So I started digging into the python schedule package, and then thought about how it might be annoying to introduce a new package and that I should revisit the cron issue. Realized cron logging wasn't work how I thought it would. Added logging specifically for this and discovered that the major issue was to remove the -t flag. I updated the cron command and instructions and it now works! Let me know if this looks good @AstridKery. Looks like there may be some merge conflicts?