Heap exhaustion on big file uploads

[mmontone]

Hi,

I'm getting heap exhaustion error when I try to upload some big file via HTTP.

           Total bytes allocated    =    1070083648
           Dynamic-space-size bytes =    1073741824
GC control variables:
   *GC-INHIBIT* = true
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
fatal error encountered in SBCL pid 230766 tid 231291:
Heap exhausted, game over.

Error opening /dev/tty: No such device or address
Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb>

When I look at backtrace, it points to: READ-BODY and READ-SEQUENCE, which to me makes sense: it is trying to load the file into memory, and that's the cause of the problem.

ldb> backtrace 10
Backtrace:
   0: fp=0x7f11a89e62e8 pc=0x53dcb83f YXORP::READ-SEQUENCE*
   1: fp=0x7f11a89e6420 pc=0x53dce878 YXORP::READ-BODY
   2: fp=0x7f11a89e6480 pc=0x53dd1d0c YXORP::HTTP-HANDLER
   3: fp=0x7f11a89e6670 pc=0x53dd2cda YXORP::PROXY-HANDLER
   4: fp=0x7f11a89e6758 pc=0x53dca064 (LAMBDA (USOCKET::CLIENT-SOCKET &REST USOCKET::ARGUMENTS) :IN USOCKET::TCP-EVENT-LOOP)
   5: fp=0x7f11a89e6838 pc=0x53dca232 (LAMBDA () :IN USOCKET::TCP-EVENT-LOOP)

Perhaps what is needed is to use FORWARD-STREAM, as it is used for websocket connections. But I don't know how to specify that for my file upload endpoint.

Please let me know if you have ideas on how I can fix this.

Thanks!

[charJe]

Thank you for bring this up. I knew this dreaded issue would surface eventually! This is not something that you can specify because it is supposed to just work. It is not as simple as just replacing read-body with forward-stream because read-body also does encoding and body filters. The encoding already uses streams, and I suppose the body filters could be changed to use streams instead of strings. I'll look into it.

[mmontone]

I thought that perhaps multipart requests could be detected and treated in a special way, just foward them "somehow" (I don't know how to do that).

Thanks for looking into this :)

[charJe]

I don't thnik treating multipart requests as special will sove the problem because the same will will pop up if you try to download a large file.

[mmontone]

You are right.

[charJe]

Greetings. It should be fixed now. Let me know how it works for you. I had to bump the version to 1 since the function signature of the filter functions changed.

Apologies for this taking so long; I got terribly distracted.

[mmontone]

Wow. This is amazing. I'll give it a try soon and let you know. Thanks!

El dom., 11 sep. 2022 23:59, charJe @.***> escribió:

Greetings. It should be fixed now. Let me know how it works for you. I had to bump the version to 1 since the function signature of the filter functions changed.

Apologies for this taking so long, I got terribly distracted.

— Reply to this email directly, view it on GitHub https://github.com/charJe/cl-yxorp/issues/2#issuecomment-1243156569, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADKPDSX7DVNYL6EAP5GV7LV52MAHANCNFSM5WM32TFQ . You are receiving this because you authored the thread.Message ID: @.***>

[charJe]

Did it work for you?

[mmontone]

I haven't tested it yet :/ I may if/when I go back to the system I was working on.