bcgov
Review list of schedulers in database
Assumptions made on frequency when developer schedulers.
- [ ] Need to review if they're appropriately timed and update if necessary (cron expressions).
- [ ] Review retry configurations.
- [ ] Review cleanup.
Schedule can be reviewed here As reference I have included the scheduled times which SFAS sends and checks for files and in the latter case, the jobs are scheduled to check for those files in relation to the scheduled time we expect them from the feds. (Relatively soon after)
However, Harry notes that it is not necessary to pick files up immediately. What matters is the file is processed before we receive the next file of the same kind.
Additional Information
- ier12-integration
- Change SIMS to check for the file every 30 minutes at :15 and :45 minutes on the hour.
- cra-process-integration
- SFAS sends file once a day @ 23:00.
- SIMS is scheduled to send the file 4x a day.
- Should we go down to once a day?
- SIMS & SFAS file are mutually exclusive, so SFAS' timing is not a consideration
- TBC [ ] Change SIMS to send file once a day @ 21:00
- cra-response-integration
- SFAS checks for response file once a day @ 06:00.
- SIMS is scheduled to check for file 4x a day.
- CRA is only sending the response file once a day, so we do not need to check 4x a day.
- [ ] Change SIMS to check for file once a day @ 08:00
- sin-validation-process-integration
- SFAS checks for response file once a day @ 04:00.
- SIMS checks for reponse file once a day @ 0:00.
- [ ] Change SIMS to check for response file once a day @ 06:00.
- sin-validation-request-integration
- SFAS sends file once a day @ 06:00.
- SIMS sends file once a day @ 0:00.
- SIMS & SFAS file are mutually exclusive, so SFAS' timing is not a consideration
- [ ] Change SIMS to send file once a day @ 06:00
- part-time-msfaa-process-integration
- SFAS sends file once a day @ 02:00
- SIMS sends file once a day @ 21:00
- NEED TO CONFIRM: Does PT MSFAA have a unique filename from SFAS? If so, SIMS & SFAS file are mutually exclusive, so SFAS' timing is not a consideration
- [ ] Change SIMS to send file once a day @ 00:00
- part-time-e-cert-integration
- SFAS sends file once a day @ 02:00
- SIMS sends file once a day @ 21:00
- [ ] Change SIMS to send file once a day @ 00:00
- part-time-feedback-integration
- SFAS checks for response file once a day @ 02:00
- SIMS checks for response file once a day @ 0:00
- [ ] Change SIMS to check for response file once a day @ 04:00
- full-time-msfaa-integration
- SFAS sends file once a day @ 03:00
- SIMS sends file once a day @ 21:00
- [ ] Change SIMS to send file once a day @ 01:00
- full-time-e-cert-integration
- SFAS sends file once a day @ 00:25
- SIMS sends file once a day @ 21:00
- [ ] Change SIMS to send file once a day @ 22:00
- full-time-feedback-integration
- SFAS checks for response file once a day @ 10:00
- SIMS checks for response file once a day @ 00:01
- [ ] Change SIMS to check for file once a day @ 12:00
- federal-restrictions-integration
- SFAS checks for file once a day @ 03:00
- SIMS checks for file once a day @ 18:00
- [ ] Change SIMS to check for file once a day @ 05:00
- full-time-disbursement-receipts-file-integration
- SFAS checks for file once a day @ 08:30
- SIMS checks file once a day @ 00:00 -[ ] Change SIMS to check for file once a day @ 10:30
- ece-process-integration
- SFAS sends file once a day @ 00:01
- SIMS sends file once a day @ 21:00
- [ ] Change SIMS to send file once a day @ 22:00
- ece-process-response-integration
- SFAS checks for file once a day @ 23:45
- SIMS checks for file once a day @ 21:00
- [ ] Change SIMS to check for file once a day @ 01:45
- fin-process-provincial-daily-disbursements-integration
- ***What file is this? ***
- THIS WILL NOT BE CHANGED AS PART OF THIS TICKET
- part-time-msfaa-process-response-integration
- SFAS checks for file once a day @ 02:00
- SIMS checks for file once a day @ 21:00
- [ ] Change SIMS to check for file once a day @ 04:00
- full-time-msfaa-process-response-integration
- SFAS checks for file once a day @ 07:15
- SIMS checks for file once a day @ 00:00
- [ ] Change SIMS to check for file once a day @ 09:15
- process-notifications
- SFAS sends file once a day @ 11:00
- Could not find this in SIMS schedule list
- sfas-integration
- Propose to Harry that we shuttle this file every 5 minutes.
- archive-applications
- SFAS does not archive applications. Harry warns against archiving, but I believe he did so under the premise that the applications would no longer be in the database, but an archived application in SIMS can still be accessed, correct?
- SIMS archives applications once a day at @14:00
- NO CHANGE REQUIRED
- assessment-workflow-enqueuer
- Not applicable in SFAS and could not find in SIMS list of scheduled jobs
- THIS WILL NOT BE CHANGED AS PART OF THIS TICKET
- assessment-workflow-queue-retry
- Not applicable in SFAS and could not find in SIMS list of scheduled jobs
- THIS WILL NOT BE CHANGED AS PART OF THIS TICKET
- student-loan-balances-part-time-integration
- SFAS checks for this file at 8:30pm everyday.
- [ ] Change SIMS to check for this file at 10:30pm everyday
Acceptance Criteria
- [ ] All times can be adjusted as a single migration.
- [ ] All times below are in PST and need to be converted to UTC.
- [ ] Change the timing of jobs as follows
- [ ] ier12-integration: Check for the file every 30 minutes at :15 and :45 minutes on the hour.
- [ ] cra-process-integration: Send file once a day @ 21:00
- [ ] cra-response-integration: Check for file once a day @ 08:00
- [ ] sin-validation-process-integration: Check for response file once a day @ 06:00.
- [ ] sin-validation-request-integration: Send file once a day @ 06:00
- [ ] part-time-msfaa-process-integration: Send file once a day @ 00:00
- [ ] part-time-e-cert-integration: Send file once a day @ 00:00
- [ ] part-time-feedback-integration: Check for response file once a day @ 04:00
- [ ] full-time-msfaa-integration: Send file once a day @ 01:00
- [ ] full-time-e-cert-integration: Send file once a day @ 22:00
- [ ] full-time-feedback-integration: Send file once a day @ 12:00
- [ ] federal-restrictions-integration: Check for file once a day @ 05:00
- [ ] full-time-disbursement-receipts-file-integration: Check for file once a day @ 10:30
- [ ] ece-process-integration: Send file once a day @ 22:00
- [ ] ece-process-response-integration: Check for file once a day @ 01:45
- [ ] part-time-msfaa-process-response-integration: Check for file once a day @ 04:00
- [ ] full-time-msfaa-process-response-integration: Check for file once a day @ 09:15
- [ ] student-loan-balances-part-time-integration:: Check for file once a day @ 22:30
- [ ] Update the above time in the schedulers wiki, adding also the missing ones and removing any that are no longer needed.
@JasonCTang Is this still relevant? If so, I don't have the details needed to answer this myself.
Also, I am assuming that if it is still relevant, it's required for PT release so I've added it to the PT release report
@andrewsignori-aot @dheepak-aot can we chat at Biz lather we need more context on what you need from the Biz. please.
@andrewsignori-aot @dheepak-aot can we chat at Biz lather we need more context on what you need from the Biz. please.
Sure @michesmith let's talk. Just to add more context, we have many schedulers executed at different times, as listed in the wiki: https://github.com/bcgov/SIMS/wiki/Schedulers. Please note, the configurations in place right now were usually a developers "best guess" and many of the "best guess" configurations happens along the years when we not even new the amount of schedulers that we should end up with.
We need to get from the Ministry (business or IMB or both):
- The expected time to have them executed.
- How frequently they should be executed? Once a day? Many times a day?
- Is there a meaningful order to have them to be executed? We can contribute from the technical perspective but maybe there is a business reason to have some orders and we need to adjust them.
- For how long should we keep the execution logs?
- Is there a preferred retry strategy to recover from failed executions?
The below is the most updated list of all schedules that we have right now. The wiki may be missing one or another.
- ier12-integration
- cra-process-integration
- cra-response-integration
- sin-validation-process-integration
- sin-validation-request-integration
- part-time-msfaa-process-integration
- part-time-e-cert-integration
- part-time-feedback-integration
- full-time-msfaa-integration
- full-time-e-cert-integration
- full-time-feedback-integration
- federal-restrictions-integration
- full-time-disbursement-receipts-file-integration
- ece-process-integration
- ece-process-response-integration
- fin-process-provincial-daily-disbursements-integration
- part-time-msfaa-process-response-integration
- full-time-msfaa-process-response-integration
- process-notifications
- sfas-integration
- archive-applications
- assessment-workflow-enqueuer
- assessment-workflow-queue-retry
- student-loan-balances-part-time-integration
@JasonCTang to take first cut at confirming when files are going to be available in SFTP and the frequency at how often they are expected.
@sslaws @andrewsignori-aot I have updated the ticket with my findings. There are a handful of jobs I could not find on the list of scheduled SIMS jobs. And there two bridge file jobs require some discussion to identify what we wish the requirement to be for timing and frequency.
Moving this to Dress/Decomp and we can put the unknown ones into a new ticket.
@JasonCTang there was a question from DEVs about PDT/PST time. The schedulers are executed in UTC and can be converted to PST or PDT. The question would be if we can stick with one of them only if we should adapt to the daylight non-daylight one. Are the ministry schedulers executed always in the current BC time meaning they would be executed with 1h difference for some period of the year?
@sslaws @andrewsignori-aot I have updated the ticket with my findings. There are a handful of jobs I could not find on the list of scheduled SIMS jobs. And there two bridge file jobs require some discussion to identify what we wish the requirement to be for timing and frequency.
The most recent list was recent in the past in the below comment: https://github.com/bcgov/SIMS/issues/1652#issuecomment-2038542626
The always updated list can always be found using the below SQL from any env.
select * from sims.queue_configurations
@JasonCTang please note that we just need times for schedulers, which means the ones with some cron expression and also that have interactions with IMB.
The below schedulers are SIMS only.
- archive-applications
- assessment-workflow-enqueuer
- assessment-workflow-queue-retry
- process-notifications
- atbc-response-integration
Moving this to Dress/Decomp and we can put the unknown ones into a new ticket.
@JasonCTang @sslaws these changes will be executed in a single DB migration. If needed, we can split the ticket but as much as possible we should have it done at once.
Add a technical AC to have the below Wiki updated also add any missing entries. https://github.com/bcgov/SIMS/wiki/Schedulers
Forgot to include these two tasks in the AC, but the ticket's already been estimated.
- Review retry configurations.
- Review cleanup. Creating a new ticket for them.
@ninosamson as mentioned in the standup meeting, as the ticket does not have an AC for sfas-integration cron schedule we are proceeding with the best guess of at :15 and :45 minutes of every hour. (e.g. runs at 10:15 and 10:45 between 10 to 11 and likewise runs twice every hour). Please confirm if this sound ok, or it should be something else?
@dheepak-aot - further info from IMB confirming the file is sent whenever there is a change, so our proposed schedule will work.