Upload fails on big files (>32MB)

[chiptus]

When uploading big files, the handler return 404 error with the following message:

{
  "err": "Invalid request payload",
  "details": "Invalid file path"
}

I checked some file sizes and it seems like 32MB is the limit.

[olljanat]

I'm just curious that what you are actually trying upload to there is you need support for that big files?

[deviantony]

Could be related to https://github.com/portainer/libhttp/issues/1

Especially https://github.com/portainer/libhttp/blob/develop/request/request.go#L44

@kendrickm could you investigate?

[deviantony]

Did a quick investigation on this, you can get more details about the issue by replacing the browsePutPayload Validate method with this one:

func (payload *browsePutPayload) Validate(r *http.Request) error {
	file, filename, err := request.RetrieveMultiPartFormFile(r, "file")
	if err != nil {
		return err
		// return agent.Error("Invalid uploaded file")
	}
	payload.File = file
	payload.Filename = filename

	path, err := request.RetrieveMultiPartFormValue(r, "Path", false)
	if err != nil {
		return agent.Error("Invalid file path")
	}
	payload.Path = path

	return nil
}

I was not able to figure out more details about this. Not a priority though, I'd rather see https://github.com/portainer/agent/issues/25 closed first.

[deviantony]

@olljanat just because we give the ability to upload files means that some users will try to upload any kind of files :-)

[olljanat]

Sure but IMO at least on first version it would good enough to give user friendly error message.

Uploading big files is anyway bit tricky as there is many times other components too which user need to take care of. Example on NGINX (which is most common reverse proxy on container world) default upload limit is 1MB.

[vast-z]

Hello, I wonder if this problem is solved or is there a solution? Thank u very much.

[deviantony]

Hi @vast-z we don't have any solution for that issue yet. As it was reported in the comments, some other components (such as reverse proxies) can also limit the request size.

[vast-z]

Thank you very much, can you tell me about the reverse proxy part of the project? I can make some modifications myself to use.

[deviantony]

@vast-z there is no reverse proxy shipped with Portainer. If you are using one (such as haproxy, nginx...) you might need to tune the configuration of the reverse proxy.

If you're not using any (accessing Portainer via IP directly) then this is an issue with the agent that we might need to investigate.

[vast-z]

Thank u very much. Yes, I didn't use a proxy and used ip directly, but the problem I encountered was that a timeout setting in portainer could not upload. I used chrome to slow down the internet speed and then uploaded a small file. After about 7-8 seconds, the connection timeout and turned off. I want to find this timeout setting and I can complete the upload.

[deviantony]

@vast-z We do not know what might be causing this issue at the moment.

See my two previous comments about this:

• https://github.com/portainer/agent/issues/28#issuecomment-429107782

• https://github.com/portainer/agent/issues/28#issuecomment-429154065