T4A - Upload File to Student Account

[CarlyCotton]

User Story As a ministry user, I need SIMS to be able to attach a PDF (T4A) to a student account. As a student, I need to be notified via email that there is a new file uploaded to my account.

Acceptance Criteria

• [ ] Create an automatic process that reviews a folder of PDF files and upload any matching ones to a student profile in SIMS.
  • [ ] Scheduled job runs weekly February - May
• [ ] Folders are in /IN/T4A/ [[This might move to /IN/]]
  • There may be more than one inside: 2024-1, 2024-2
• [ ] Each folder will have 1 to MANY pdf files in it with a filename that corresponds to a student's SIN
  • "399000039.pdf"
• [ ] If SIMS has had a verified SIN that matches the filename, upload the file to that student's profile. Student can have multiple verified SINs.
  • The file name should be updated to "[folder name - e.g. "2024-1"]-T4A-yyyy-mm-dd.pdf"
  • This should be done as 'system-user'
  • Should still trigger the regular upload notification email to the student
• [ ] If SIMS does not have a corresponding Student Profile with a verified SIN that matches the filename ignore the file and archive
• [ ] On subsequent job executions, if there is a new T4A PDF for a student add new PDF

[ninosamson]

Moving to 3.X as T4A is uploaded in new year (2026).

[ninosamson]

Confirm business requirements on revisions with SABC FIN; confirm folder naming with Harry

[ninosamson]

Critical Complete 2.

[andrewsignori-aot]

@CarlyCotton @ninosamson, we do not currently have a generic IN folder, which means that we need to either create a configuration for the T4A (e.g., T4A_FOLDER). Is it ok?

[andrewsignori-aot]

@CarlyCotton @ninosamson is it possible to have the below AC subitem adjusted to a final decision, or do we need to wait for next week?

• [ ] Folders are in /IN/T4A/ [[This might move to /IN/]] - There may be more than one inside: 2024-1, 2024-2

[andrewsignori-aot]

@CarlyCotton do we need to have some extra validation in case the same valid SIN is found associated with multiple students? If yes, should we just raise it as a warning that will then generate an alert on sysdig?

[andrewsignori-aot]

@CarlyCotton, when the files are saved, we generate a group_name on the DB record that ends up being displayed below. Can we have an AC with the expected text to be displayed (e.g., T4A)?

[CarlyCotton]

@andrewsignori-aot updated to add ac:

Document purpose for these uploaded files should be "T4A"

[CarlyCotton]

@andrewsignori-aot

do we need to have some extra validation in case the same valid SIN is found associated with multiple students? If yes, should we just raise it as a warning that will then generate an alert on sysdig?

Warning that will generate an alert is fine.

is it possible to have the below AC subitem adjusted to a final decision, or do we need to wait for next week?

Waiting on Harry's response net week

we do not currently have a generic IN folder, which means that we need to either create a configuration for the T4A (e.g., T4A_FOLDER). Is it ok?

They will be within IN/T4A . Not sure if it needs to be configuration, I don't see it needing to be changed but wouldn't be harmful to have as configuration if that's nicer.

[andrewsignori-aot]

@CarlyCotton, related to the below AC, as we talked today during dev sync up, can we have the job running daily instead of weekly?

• [ ] Scheduled job runs weekly February - May

[andrewsignori-aot]

@ninosamson @CarlyCotton because the files can take a bit longer, for an upcoming PR in this ticket I can add the below progress to be increment once every file is completed.

[andrewsignori-aot]

@CarlyCotton can we have some ACs related to logs and SIN information retention during processing?

For SIN on logs we can have something like below.

• [ ] SIN number should not be persisted in the logs since it is sensitive information.

Related to the retention period of the completed queues that will have the SIN on its payload, we talked about removing them immediately on completion. If this is still the idea, can we have an AC added? If not, is there some amount of time that we would like to keep the completed jobs?

[andrewsignori-aot]

@CarlyCotton question raised during PR review, should we have a specific "file_origin" for T4A?

[andrewsignori-aot]

@CarlyCotton, for archiving, the files will be moved to the root of the archived files following the pattern from other integrations. Is there any need to have a special archive folder for T4A files?

[andrewsignori-aot]

Demo

@CarlyCotton, demo information prepared so far. If further scenarios are required, I can prepare them. Schedulers wiki updated with the new scheduler.

New scheduler and queue deployed to DEV and scheduled to run next February.

t4a-upload-emqueue

Tested reading 39,999 files from the SFTP and creating 400 queues in t4a-upload.

[2025-12-17 00:42:29]INFO: Log details
[2025-12-17 00:42:14]INFO: Processing queue t4a-upload-enqueuer, job ID repeat:c60eb18336ea5f2eb4a82b3897c56ea7:1769950800000.
[2025-12-17 00:42:14]INFO: Checking T4A files to be enqueued for processing.
[2025-12-17 00:42:14]INFO: Max file uploads per batch configured as 100.
[2025-12-17 00:42:15]INFO: Found T4A directories: IN/T4A/2025-LOAD-TEST.
[2025-12-17 00:42:15]INFO: Processing T4A files in IN/T4A/2025-LOAD-TEST.
[2025-12-17 00:42:29]INFO: Found 39999 files in IN/T4A/2025-LOAD-TEST, in 13538.72ms.
[2025-12-17 00:42:29]INFO: Time to queue files, 142.61ms.
[2025-12-17 00:42:29]INFO: t4a-upload-enqueuer, job ID repeat:c60eb18336ea5f2eb4a82b3897c56ea7:1769950800000, finished.

CPU and memory usage after the import

Note: there were no students associated with the fake SIN numbers, hence the full process was not executed for the below ones.

Tested with just a few files for existing students with SIN matches

Jack

• Current SIN: 747531358
• Past SIN: 426221628
• Student ID: 664

Franky

• Current SIN: 225376896
• Student ID: 171

Notification

[andrewsignori-aot]

@ninosamson, we are currently using the Bull Queues framework which was a pretty solid and recommended framework was it was adopted, but it is now in maintenance mode.

The long-term solution would migrate to BullMQ. It would require some analisys since it is a similar framewrok but a direct replacement.

[ninosamson]

Thanks @andrewsignori-aot - I'm pulling the BullMQ ticket into grooming for discussion.

[andrewsignori-aot]

@CarlyCotton @ninosamson Can we please have the following environment variables added to the release instructions?

• T4A_FOLDER: IN/T4A
• T4A_ARCHIVE_FOLDER: IN/Archive

[andrewsignori-aot]

@ninosamson, we have a mix of GitHub env and secrets where many secrets are not sensitive information and should be moved to regular GitHub vars. Can we have a ticket to have it organized? In the past, GitHub did not have the concept of env vars, only secrets and that was the only reason to have non-sensitive values created as a secret.

Recommended ACs.

• [ ] Identify, along with the GitHub secrets, any entry that is not considered sensitive.
• [ ] Create a list and review it with a Ministry representative (IMB).
• [ ] Move them from secrets to vars and adjust their references in GitHub actions, including parameters passed between workflows.

[andrewsignori-aot]

@CarlyCotton @ninosamson, as agreed, we will change the "removal on completed" to adjusted to keep them for 7 days.

[andrewsignori-aot]

@CarlyCotton @ninosamson the last PR was merged with the changes in cleanup to keep the queues for 7 days.

[CarlyCotton]

@andrewsignori-aot awesome! Closing this one again.