Archive SFTP files

[JasonCTang]

Describe the task Archive files on the SFTP server.

• When reading files uploaded by external systems, archive the file using the following logic. (Instead of deleting which is what happens now)

  • If file is proccessed succesffully, move them to an ./Archive/ subfolder which will be in the same folder as the file being processsed.
  • If a non-sequential file fails processing, move them to an ./Archive/ subfolder which will be in the same folder as the file being processed.
  • If a sequential file fails processing, do not archive.

• These change applies to all files being read:

  • Institution files
  • MSFAA files
  • Federal files
  • Bridge files
  • And others, if any.

Acceptance Criteria

• [ ] Investigate and identify which files MUST be processed sequentially and which ones do not.
• [ ] Whenever a file (sequential or non-sequeuntial) is processed successfully, move the file into the ./Archive/ subfolder located in the same folder as the file being processed.
• [ ] Whenever a non-sequential file fails processing, move file into the ./Archive/ subfolder located in the same folder as the file being processed.
• [ ] Whenever a sequential file fails processing, do not move file.
• [ ] Provide a list of sequentially executed ones, non-sequentially executed ones, and delete behavior (e.g. deleted on error or not) defined during development.
• [ ] E2E tests
  • [ ] Adjust the E2E tests checking for the delete operation to start checking for the "move" operation.

[JasonCTang]

Describe the task Instead of deleting response files on the SFTP server after they're read into SIMS, copy them to an ./Archive/ subfolder. The location of this archive subfolder will be in the same folder as the file being read. This change applies to all files being read:

• Institution response files
• Federal response files
• Bridge files

Acceptance Crteria

• [ ] Whenever a file is successfully read into SIMS, move the file into the ./Archive/ subfolder located in the same folder as the file being read.

[ninosamson]

Keeping ticket open to clarify/discuss wiki and sequential processing of files.

[ninosamson]

Let's close this ticket and open another ticket to track monitoring, errors and operational processes with regards to files.

[andrepestana-aot]

Clarifying the sequential processing of files @ninosamson @JasonCTang I'd like to highlight that full-time-feedback-integration and part-time-feedback-integration are actually not sequential but still, they don't archive the files on error. That is following the previous logic for deletion. The wiki was updated accordingly https://github.com/bcgov/SIMS/wiki/Schedulers The file handling will be analyzed on https://app.zenhub.com/workspaces/student-information-management-system-5fce9df5aa1b45000e937014/issues/gh/bcgov/sims/3666