asgi_ipc icon indicating copy to clipboard operation
asgi_ipc copied to clipboard

MemoryError on large uploads

Open scardine opened this issue 6 years ago • 1 comments

Looks like Daphne will choke on large POST requests (in my case while uploading a large file).

out 16 16:48:12 icr daphne[29020]: 2017-10-16 16:48:12,424 ERROR    Traceback (most recent call last):
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/daphne/http_protocol.py", line 174, in process
out 16 16:48:12 icr daphne[29020]:     "server": self.server_addr,
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/core.py", line 68, in send
out 16 16:48:12 icr daphne[29020]:     time.time() + self.expiry,
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/store.py", line 116, in append
out 16 16:48:12 icr daphne[29020]:     value.setdefault(name, []).append((item, expiry))
out 16 16:48:12 icr daphne[29020]:   File "/usr/lib/python3.6/contextlib.py", line 89, in __exit__
out 16 16:48:12 icr daphne[29020]:     next(self.gen)
out 16 16:48:12 icr daphne[29020]:   File "/var/app/virtualenvs/ev/lib/python3.6/site-packages/asgi_ipc/store.py", line 64, in mutate_value
out 16 16:48:12 icr daphne[29020]:     pickle.dump(value, self.mmap, protocol=2)
out 16 16:48:12 icr daphne[29020]: MemoryError

I'm using Python 3.6.1 (64bit) on Ubuntu 17.04 with 8G RAM. Uploaded file is 1.6GB.

Looks like a problem with pickle, is there any way to use other pickle-compatible serializer that is better for larger objects?

scardine avatar Oct 16 '17 19:10 scardine