paygate icon indicating copy to clipboard operation
paygate copied to clipboard

transfers/inbound: clean up zero-byte files

Open adamdecaf opened this issue 3 years ago • 1 comments

PayGate Version: v0.9.0-dev

During periods of no activity some vendor systems create zero-byte files that PayGate needs to process and delete. These files will not properly parse as incoming/returned files and need to just be deleted, however zero-byte files might exist during the middle of a transfer from the ODFI's side. We don't want to delete these files right away, but after they've been sitting on the remote machine for a while (e.g. 10m by default).

Similar to KeepRemoteFiles: <boolean> I think we want a configuration to remove these outside of that option.

RemoveZeroByteFilesAfter: <duration> # default 10 minutes

These files would be removed as part of transfers/inbound's cleanup.go file: https://github.com/moov-io/paygate/blob/v0.9.0-dev1/pkg/transfers/inbound/cleanup.go

adamdecaf avatar Oct 16 '20 18:10 adamdecaf

There's a bug with the current implementation of this. When we download a file onto the local filesystem the created and modified times are set to "now", but the file on the SFTP server would have been created "in the past." The logic expects to stat the local file and have it represent the old ctime.

We need to either write the ctime/mtime from the server (only viable on unix systems) or stat the remote file and decide then. I believe the FTP library we use doesn't support STAT, but our sftp library does. We could leverage just the SFTP side for now.

adamdecaf avatar Nov 23 '20 21:11 adamdecaf