UploadFile(Stream) calls SIZE when overwrite behavior is set to NoCheck

[j-childers]

FTP OS: Any

FTP Server: Any

Computer OS: Any

FluentFTP Version: 37.0.2

We have a use case where we send files via FTP that are guaranteed to have unique names, and want to send them as quickly as possible. The destination only supports FTP. So we desire as few round-trips as possible. While optimizing the transfer we noticed that even when we tell it not to check for any existing file, the SIZE command is still sent before starting the transfer.

It appears that OpenWrite checks the remote file size because UploadFileInternal doesn't set the fileLen parameter to a non-zero value. It would be nice to have a way to prevent this by exposing fileExistsKnown as a parameter to UploadFile, passing this to UploadFileInternal, and setting the remoteFileLen to the local stream length if existsMode is NoCheck, fileExists is false, and fileExistsKnown is true.

[robinrodricks]

I guess you are asking for a low level method when you know the filesize. We have such methods (UploadFileInternal) which you can access using reflection or by subclassing the FtpClient class.

[j-childers]

Not exactly. It's not about knowing the size or not. We know the file does not exist at all (i.e. there is no size). Subclassing is an option, and when the project picks up again next week, I'll probably go that route in order to shave a few milliseconds off.

Just thought it might benefit others (however slightly) since it seems inefficient to send a SIZE command when the caller explicitly says not to check for the file's existence.

[robinrodricks]

Just thought it might benefit others (however slightly) since it seems inefficient to send a SIZE command when the caller explicitly says not to check for the file's existence.

Then its probably a bug. Which enum value were you using for the RemoteExists parameter?

[j-childers]

Here's how we're calling it:

_FtpClient.Upload(stream, $"{remoteFolderName}/{remoteFileName}", FtpRemoteExists.NoCheck, false, uploadProgressAction)

[FanDjango]

I have researched this for V40:

If you do not specify NoCheck there will be two SIZE commands. One belonging to FileExists, one belonging to the prelude of OpenWrite - GetFileSize.

If you do specify this, there will be "just" one SIZE command, belonging to the prelude of OpenWrite - GetFileSize.

And I agree that this can (and will) be optimized.

[FanDjango]

It would be nice to have a way to prevent this by exposing fileExistsKnown as a parameter to UploadFile, passing this to UploadFileInternal, and setting the remoteFileLen to the local stream length if existsMode is NoCheck, fileExists is false, and fileExistsKnown is true.

Yes

[FanDjango]

Actually, after some debugging and looking at the current code, I come to the following conclusion:

fileExistsKnown is irrelevant in the case that NoCheck is true and needs not be propagated in the fashion you described.

It is sufficient to set remoteFileLen to remoteFileLen, i.e.

...
				// calc local file len
				var localFileLen = fileData.Length;

				//
				if (existsMode == FtpRemoteExists.NoCheck) {
					remoteFileLen = localFileLen;
				}
...

A file transfer then looks like this:

>         UploadFile("D:\temp\test1", "/home/mike/test1", NoCheck, False, None)
>         OpenWrite("/home/mike/test1", Binary)
>         OpenActiveDataStream(PORT, "STOR /home/mike/test1", 0)
PORT 192,168,1,147,31,146
200 PORT command successful
STOR /home/mike/test1
150 Opening BINARY mode data connection for /home/mike/test1
226 Transfer complete

Note: Setting NoCheck effectively tells FluentFTP: No need to check for existence as it does not exist

The internally used fileExistsKnown is geared towards telling UploadFileInternal(...) that the current file is from a list of existing files - it is a separate indicator to not perform an existence check in addition to existsMode which is given by the user.

Conversely, if existsMode is not set to NoCheck, an existance check will take place, using either (in most cases) a SIZE command, an MDTM command or even a NLST name listing.

Later on OpenWrite will again try to get the size - thus we have the observed phenomenon of the two SIZE commands. Well - that can be optimised too, for the case that the file does not exist. and also for the case that an overwrite is requested.

There will be a PR for this soon. @j-childers Are you still out there and ready to play with this when it gets merged?

[j-childers]

So sorry to have missed getting back to you! I will give this a try later this week, but thanks for putting up the PR. The change you committed looks good, though.

[j-childers]

Got in and looked this evening. The change works as expected. Thanks again.