High RAM cache usage when uploading files

[dokicro]

What steps will reproduce the problem?

Upload a large file (10GB +)

What is the expected result?

That upload will be successful

What do you get instead?

I get HIGH Ram cache usage

After the last chunk nginx responds with 502 and that's because files cant be merged into one file since there is no RAM left to do that.

502 error happens on $save->isFinished()

Additional info

Q	A
This Package Version	1.5.2
Laravel Framework Version	9.51.0
PHP version	8.1
Operating system	Ubuntu 22

[pionl]

How large chunks you are using?

[dokicro]

Chunks are 1mb, and the 502 was resolved by changing max_execution_time but still I am not sure if it's expected that the whole RAM gets filled with catch?

[nerg4l]

Could you share a code example? I can try to reproduce the issue.

[dokicro]

Hi,

I was debugging it more and so when you start the upload and open htop yellow bar starts increasing which indicates cache:

What is maybe unique to my setup is that I am using laravel horizon (https://laravel.com/docs/10.x/horizon) inside laravel sail (https://laravel.com/docs/10.x/sail)

I don't know if this is the cause of this issue, maybe this is expected behavior...

The code is from your example:

public function largeFile(FileReceiver $receiver)
    {
        // check if the upload is success, throw exception or return response you need
        if ($receiver->isUploaded() === false) {
            throw new UploadMissingFileException();
        }
        // receive the file
        $save = $receiver->receive();

        // check if the upload has finished (in chunk mode it will send smaller files)
        if ($save->isFinished()) {
            // save the file and return any response you need
            return $this->storageService->saveFile($save->getFile());
        }

        // we are in chunk mode, lets send the current progress
        /** @var AbstractHandler $handler */
        $handler = $save->handler();

        return response()->json([
            'done' => $handler->getPercentageDone(),
        ]);
    }

[pionl]

The code for merging is using buffer so it should not get so large.

I will try it but thanks for pointing sail/horizon.

[dokicro]

If you need any help with setting that up please let me know and I can help you :)

[dokicro]

BTW. One more thing rams get full while uploading the files not then merging starts...