Runtime error: invalid memory address or nil pointer dereference when trying to upload file bigger than 32 MB

[sky-nunomarcos]

Hi There, I'm facing when trying to upload files bigger than 32MB, is there any configuration I can apply to solve this issue?

Launching HTTP Server Docker Image: ➜ temp docker run -it --rm -p 8000:8000 -v $PWD:/app/public --name gohttpserver codeskyblue/gohttpserver --upload --delete 2020/12/14 19:39:42 httpstaticserver.go:66: root path: /app/public/ 2020/12/14 19:39:43 main.go:164: plistproxy: "https://plistproxy.herokuapp.com/plist" 2020/12/14 19:39:43 main.go:212: listening on ":8000", local address http://172.17.0.2:8000 2020/12/14 19:39:44 httpstaticserver.go:78: Started making search index 2020/12/14 19:39:44 httpstaticserver.go:80: Completed search index in 3.395757ms 2020/12/14 19:39:55 main.go:55: 172.17.0.1 - POST 200 / 2020/12/14 19:40:02 httpstaticserver.go:239: Move /tmp/multipart-236259721 -> /app/public/temp_file 2020/12/14 19:40:02 httpstaticserver.go:252: Handle upload file: 2020/12/14 19:40:02 server.go:3085: http: panic serving 172.17.0.1:62756: runtime error: invalid memory address or nil pointer dereference goroutine 13 [running]: net/http.(*conn).serve.func1(0xc000074320) /usr/local/go/src/net/http/server.go:1795 +0x139 panic(0x933ee0, 0xe2afd0) /usr/local/go/src/runtime/panic.go:679 +0x1b2 main.(*HTTPStaticServer).hUploadOrMkdir(0xc0000d4510, 0xa96a40, 0xc00008c2c0, 0xc0000d3100) /app/gohttpserver/httpstaticserver.go:253 +0x680 net/http.HandlerFunc.ServeHTTP(0xc0000f0b70, 0xa96a40, 0xc00008c2c0, 0xc0000d3100) /usr/local/go/src/net/http/server.go:2036 +0x44 github.com/gorilla/mux.(*Router).ServeHTTP(0xc0000b9ea0, 0xa96a40, 0xc00008c2c0, 0xc0000d3100) /go/pkg/mod/github.com/gorilla/[email protected]/mux.go:162 +0x104 main.(*HTTPStaticServer).ServeHTTP(0xc0000d4510, 0xa96a40, 0xc00008c2c0, 0xc0000d2e00) /app/gohttpserver/httpstaticserver.go:97 +0x50 github.com/codeskyblue/go-accesslog.(*LoggingHandler).ServeHTTP(0xc0000ed7a0, 0xa97140, 0xc000124000, 0xc0000d2e00) /go/pkg/mod/github.com/codeskyblue/[email protected]/accesslog.go:162 +0x3cf net/http.(*ServeMux).ServeHTTP(0xe3d320, 0xa97140, 0xc000124000, 0xc0000d2e00) /usr/local/go/src/net/http/server.go:2416 +0x1bd net/http.serverHandler.ServeHTTP(0xc0001242a0, 0xa97140, 0xc000124000, 0xc0000d2e00) /usr/local/go/src/net/http/server.go:2831 +0xa4 net/http.(*conn).serve(0xc000074320, 0xa98600, 0xc00001e300) /usr/local/go/src/net/http/server.go:1919 +0x875 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2957 +0x384

When I try to run the following cmd commands: ➜ ~ mkfile -n 32m temp_file ➜ ~ curl -F file=@temp_file localhost:8000 {"destination":"/app/public/temp_file","success":true} ➜ ~ mkfile -n 33m temp_file ➜ ~ curl -F file=@temp_file localhost:8000 curl: (52) Empty reply from server

Can you help me out with this, please?

Thanks, Nuno Marcos

[inmanturbo]

Same issue. Have you found a fix for this?

[MartinHerren]

Same issue here using the docker image from docker hub. Fixed by rolling back to the previous version we had. As there is only the latest one on docker hub, i don't know if the previous one is still available somewhere. Luckily we still had it on our server. (image dated march 2019...)

[aeoluswing]

The real cause of this problem is if the upload file size larger than 32MB, it will be processed as a temporary file first.After the file upload process is executed, the server will call the os.Rename method to rename to the target file name.The panic mentioned in this issue occurred during this process.Please pay attention to the following code snippet:

if osFile, ok := file.(*os.File); ok && fileExists(osFile.Name()) {
    tmpUploadPath := osFile.Name()
    osFile.Close() // Windows can not rename opened file
    log.Printf("Move %s -> %s", tmpUploadPath, dstPath)
    copyErr = os.Rename(tmpUploadPath, dstPath) // Panic occurs when the file is larger than 32MB
} else {
    dst, err := os.Create(dstPath)
    if err != nil {
        log.Println("Create file:", err)
	http.Error(w, "File create "+err.Error(), http.StatusInternalServerError)
	return
    }
    _, copyErr = io.Copy(dst, file)
    dst.Close()
}

I will submit a PR for fixing, but currently this modification allows me to upload files larger than 32MB without errors.

[aeoluswing]

Noticed that a fix has been releasedIssue98.However, large size files might still cause poor performance in the copy process.