StreamWrapper - handling large file uploads

[djoos]

Hi,

in order for us to make use of Gaufrette, we needed to find a way round some limitations, in particular when handling large files. We came up with https://github.com/escapestudios/EscapeGaufretteExtension as a "extension"/work-around (in short: we've added a TempFileBuffer which uses a temp file to write to rather than keeping the content in memory) and a small change to the way the OpenCloud adapter writes an object (in short: use a DataObject's Create-method with a filename when the content is a file or the DataObject's setData method when the content is not a file).

It would be great to see this or something similar implemented into Gaufrette, as then large file uploads can be taken care of in the great Gaufrette-way!

Thanks in advance for your feedback!

Kind regards, David

[djoos]

Top tip when dealing with large file uploads to OpenCloud: make sure to define RAXSDK_TIMEOUT with a higher-than-default (default = 30) value.

eg.

define('RAXSDK_TIMEOUT', 1800);

Hope this helps!

Kind regards, David

[geoffreytran]

I am having the same issue with handling large file uploads. All adapters that do support resource write() call Util/Size which uses mb_strlen to determine content size. This should be avoided.

[wysow]

@djoos Is this still relevant?

[djoos]

Hi @wysow,

yes, the issue is still relevant: AFAIK the file content is still kept in memory in the bundle when not using our tempfile workaround (see above)... This is an issue when handling (very) large file uploads...

Hope this helps!

Kind regards, David

[wysow]

@djoos As it seems you managed to make it work in your "extension" https://github.com/escapestudios/EscapeGaufretteExtension, is it possible for you to make a PR based on this?

[djoos]

Definitely, I'll keep you updated!

[SrgSteak]

Is there working build available now?

[djoos]

Hi @SrgSteak,

I haven't got round to starting the work of getting our "extension" into Gaufrette...

However, I haven't forgotten this thread!

Kind regards, David

[Uplink03]

I've taken the liberty to integrate the temp file buffer functionality from EscapeGaufretteExtension into my own fork (but don't use this - see next comment): https://github.com/Uplink03/Gaufrette/tree/enhance/escape-gaufrette

I didn't take the OpenCloud changes.

I didn't put too much thought into it, so now the InMemoryBuffer isn't used and there's no option to select it. But maybe it helps somebody. If you review the changes and find them good for a PR, I'll make a PR.

[Uplink03]

UPDATE: DON'T USE THIS CODE. It's introduces an incompatible API change. Just check out what happens in TempFileBuffer's flush method: the content is the temp file's filename. Without reviewing the OpenCloud changes, I'm gonna say that they address this behaviour, but if you go and use, say, S3, you get the temp file's filename instead of the content.

While researching this I noticed that I can pass fopen handles to filesystem->write, so I can replace all file_get_contents with fopen and get rid of the memory issues. Both AwsS3 and Local adapters accept this usage, and I would suggest it gets documented and expected across the board from all other adapters.

This also means I don't need to use streams at all.

A few days ago I said:

I've taken the liberty to integrate the temp file buffer functionality from EscapeGaufretteExtension into my own fork: https://github.com/Uplink03/Gaufrette/tree/enhance/escape-gaufrette

I didn't take the OpenCloud changes.

I didn't put too much thought into it, so now the InMemoryBuffer isn't used and there's no option to select it. But maybe it helps somebody. If you review the changes and find them good for a PR, I'll make a PR.