OverflowError and MemoryError when uploading files over 3GB

[rj-d2]

im having problems uploading files above 3 GB i get OverflowError and MemoryError running gofile 0.3.0

tried to upload a 3.4 GB file OverflowError:

Uploading progress: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- Traceback (most recent call last): File "/home/REDACTED/.pyenv/versions/3.9.1/bin/gofile", line 8, in sys.exit(gofile.main()) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 183, in main gofile_upload( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 89, in gofile_upload upload_resp = upload(file, best_server, folder_id).json() File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 36, in upload resp = requests.post( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/http/client.py", line 1049, in _send_output self.send(chunk) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/http/client.py", line 971, in send self.sock.sendall(data) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/ssl.py", line 1204, in sendall v = self.send(byte_view[count:]) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/ssl.py", line 1173, in send return self._sslobj.write(data) OverflowError: string longer than 2147483647 bytes

tried to upload a 5.2 GB file MemoryError:

Uploading progress: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% -:--:-- Traceback (most recent call last): File "/home/REDACTED/.pyenv/versions/3.9.1/bin/gofilepy", line 8, in sys.exit(gofile.main()) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 183, in main gofile_upload( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 89, in gofile_upload upload_resp = upload(file, best_server, folder_id).json() File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/gofilepy/gofile.py", line 36, in upload resp = requests.post( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/sessions.py", line 573, in request prep = self.prepare_request(req) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/sessions.py", line 484, in prepare_request p.prepare( File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/models.py", line 371, in prepare self.prepare_body(data, files, json) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/models.py", line 556, in prepare_body (body, content_type) = self._encode_files(files, data) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/requests/models.py", line 201, in _encode_files body, content_type = encode_multipart_formdata(new_fields) File "/home/REDACTED/.pyenv/versions/3.9.1/lib/python3.9/site-packages/urllib3/filepost.py", line 90, in encode_multipart_formdata body.write(data) MemoryError

any idea?

[Alyetama]

Can you run this command in a terminal window, then post the output here?

top -l 1 -s 0 | grep PhysMem

[rj-d2]

thx for your fast answer,

your command isnt working:

top -l 1 -s 0 | grep PhysMem
top: unknown option 'l'

im usually running gofile on a shared seedbox which has great speeds and is usually running great, maybe they limited the ram somehow.

so i tried to upload some files from my desktop pc 16GB ram running linux mint 21 and noticed my ram is used 100% while uploading whith my poor 20mbit/s internet connection, even the 2GB swap memory is used 100% is it normal that the upload is using so much memory? i tried uploading a 5GB file, is this expected behaviour?

thx for your help

[rj-d2]

would it be somehow possible to reduce the memory needed to upload? i would not care to loose some upload speed as long as the uploads finishes.

[BrianJ20]

Am seeing the same thing as the guy above. A 1gb file uploads no problem, little memory use. A 10gb file uses 100% memory(32gb) but never actually uses any upload bandwidth.

The computer is unusable while this goes on.

This is on windows 11 using Python 3.10.5

[Alyetama]

This probably has to do with how the requests library handle the request. I will resolve it in the next PR this week.

[github-userx]

Message ID: @.***>https://github.com/Sushrut1101/GoFile-Upload

Is written in Bash and uses curl, jq

But only allows upload of single files.

The great multi file hosts Upload Tool „Anypaste“ written in bash also supports gofile.io upload among others:

https://github.com/markasoftware/anypaste  /  https://anypaste.xyz/#plugins

Maybe these can help with the fix.. Looking forward to the fix of this issue. Many thanks!