Is there any future plan to support HTTP Server side File Upload?

[Coldzer0]

Hello @synopse

I'd like to know if there are any plans to support HTTP server file upload.

I have a client code that does a THttpMultiPartStream and sends the stream to the server. But in my testing, there's no clear way in mormot2 to handle this kind of file upload, especially if the files are big (> 2GB).

If there are no plans to support this in the near future, Can you help by giving instructions on how to approach this in mormot2? ( Like how to handle the data by putting it directly into a file stream and not keeping it in memory without messing with the source code of mormot itself )

mORMot2 code is vast, and even after a couple of years of working with it, I still find new stuff and learn from it ^_^

[synopse]

It is indeed not yet supported.

But it is planned in the close future, with the upcoming proxy and reverse proxy features of our async server. We need this, because nginx is no option on Windows, and our async server using IOCP seems to be pretty fast, so we will continue to enhance it.

So stay tuned, we will support it, I hope before the end of this year. I guess you can close this issue, and open it back in a few weeks if you see no progress in that direction.

[synopse]

Note that all the internal logic for using a TStream instead of a RawByteString is already there. We just need a mean to trigger it properly.

Perhaps in OnBeforeBody or - perhaps better - with a route registration. What do you think?

[Coldzer0]

I was writing notes about building something using OnBeforeBody and Parsing the Headers. And then we know we have the Content-Type to detect the multipart file upload and the Content-Length. And for sure, the boundary string means that we know this is the end of file data and that we have started a new file.

var 
  Files : THttpMultiPartStream;  
Begin
  Files := THttpMultiPartStream.Create;
  Files.AddFile('files', 'F:\UploadTest\hello.txt'); // Contains 'Hello'
  Files.AddFile('files', 'F:\UploadTest\world.txt'); // Contains 'World'
  Files.Flush; 
  ...
End;

Headers for it on the server side look like this

Cache-Control: no-cache
Connection: Close
Pragma: no-cache
Content-Type: multipart/form-data; boundary=oKhwAf29zfOYjwQy
Accept: */*
User-Agent: Mozilla/5.0 (Win x64; mORMot) WH/2 FileUpload_Debug64
Content-Length: 368
Host: localhost

The content will be

--oKhwAf29zfOYjwQy
Content-Disposition: form-data; name="files"; filename="hello.txt"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary

Hello
--oKhwAf29zfOYjwQy
Content-Disposition: form-data; name="files"; filename="world.txt"
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: binary

World
--oKhwAf29zfOYjwQy--

Considering that the content will be sent in chunks, a little bit of parsing would be required to detect when a file ends.

Adding a route registration for file upload is also a great idea.

[synopse]

My guess is that our TStream support should be separated from mullti-part decoding, because

• Huge content could be sent without multi-part, but e.g. with or without chunking
• A single TStream is not a good candidate to split several multi-part files content

Most advanced reverse proxy servers do use a temporary file as output. Letting a callback generate a TStream from the received input may be a good similar solution.

[Coldzer0]

My guess is that our TStream support should be separated from mullti-part decoding, because

• Huge content could be sent without multi-part, but e.g. with or without chunking

What about saving the entire data into a temp file, and then we do the parsing? I know parsing in memory is way faster, but we can handle all cases this way.

• A single TStream is not a good candidate to split several multi-part files content

That's true, but when a multi-file begins to be sent, it will be sent in one request; that's why I thought about the temp file and then parsing things. Of course only if needed data was found to be parsed.

Most advanced reverse proxy servers do use a temporary file as output. Letting a callback generate a TStream from the received input may be a good similar solution.

In this case, route registration for file upload is necessary so that any file upload requests will be ignored and the connection terminated. This can prevent a lot of DDOS attacks. But still, the user is the one responsible for handling all the cases.

[synopse]

Yes, the TStream would be a file, either with a name supplied by the callback, or a temporary name.

[Coldzer0]

It is indeed not yet supported.

But it is planned in the close future, with the upcoming proxy and reverse proxy features of our async server. We need this, because nginx is no option on Windows, and our async server using IOCP seems to be pretty fast, so we will continue to enhance it.

So stay tuned, we will support it, I hope before the end of this year. I guess you can close this issue, and open it back in a few weeks if you see no progress in that direction.

@synopse

Any updates on this feature ^_^

[synopse]

Currently looking into that.

It sounds like if we could have chunked + multipart. https://stackoverflow.com/questions/13969403/chunked-transfer-and-multipart-http

So since we already have chunked support with a destination TStream, I will need to make something like a new THttpMultiPartServerStream class. And then implement something close to what Nginx offers with client_body_in_file_only clean from this THttpMultiPartServerStream class.

[Coldzer0]

Hello @synopse

I hope you’re having a good week ^_^ I just want to know if there will be any progress to this anytime soon.

Thanks.