Clamav file size limit - throw custom error

[hanoj-budime]

For such errors - Clamav file size limit

Are you throwing with an exact error message?

---

Expecting

Somehow it should return the same error message to us

This WARNING message should return to ClamScan level so that we can understand those errors - we can throw custom based on your error message info.

[hanoj-budime]

Hi @kylefarris,

Please can you help me

[hanoj-budime]

Hi @SaltwaterC, @genio, @ngraef

Please can you help me it's urgent to me 😊

[benzino77]

I'm observing the same situation (clamd v0.104 and clamd v0.105). What I can say is that sometimes it is rejected on socket.error and sometimes it is resolved with NodeClamError:

Reject example:

node-clam: Provided stream is readable.
node-clam: Attempting to establish socket/TCP connection for "scanStream"
node-clam: using remote server: 127.0.0.1:3310
node-clam: Received final data from stream.
node-clam: The input stream has dried up.
node-clam: _initSocket Socket/Host connection failed: Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16) {
  errno: 'EPIPE',
  code: 'EPIPE',
  syscall: 'write'
}
node-clam: scanStream Error emitted from ClamAV socket:  Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16) {
  errno: 'EPIPE',
  code: 'EPIPE',
  syscall: 'write'
}
node-clam: _initSocket Socket/Host connection closed.
node-clam: scanStream ClamAV socket has been closed! true

and here is Resolve with NodeClamError example:

node-clam: Provided stream is readable.
node-clam: Attempting to establish socket/TCP connection for "scanStream"
node-clam: using remote server: 127.0.0.1:3310
node-clam: Received final data from stream.
node-clam: The input stream has dried up.
node-clam: scanStream Received output from ClamAV Socket.
node-clam: scanStream ClamAV is done scanning.
node-clam: scanStream Raw Response:  INSTREAM size limit exceeded. ERROR
node-clam: Error Response:  INSTREAM size limit exceeded.
node-clam: File may be INFECTED!
CUSTOM DEBUG scanStream before resolve:  NodeClamError: An error occurred while scanning the piped-through stream: INSTREAM size limit exceeded.
    at NodeClam._processResult (/home/benzino/Dokumenty/clamav-rest-api/node_modules/clamscan/index.js:781:20)
    at Socket.<anonymous> (/home/benzino/Dokumenty/clamav-rest-api/node_modules/clamscan/index.js:2283:45)
    at Socket.emit (events.js:326:22)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  data: { error: 'INSTREAM size limit exceeded. \u0000' },
  date: 2022-07-27T10:07:14.898Z
}
node-clam: _initSocket Socket/Host connection failed: Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16) {
  errno: 'EPIPE',
  code: 'EPIPE',
  syscall: 'write'
}
node-clam: scanStream Error emitted from ClamAV socket:  Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:94:16) {
  errno: 'EPIPE',
  code: 'EPIPE',
  syscall: 'write'
}
node-clam: _initSocket Socket/Host connection closed.
node-clam: scanStream ClamAV socket has been closed! true

[benzino77]

I have investigated it a little bit more, and it looks like it can be a problem on clamscan side (don't know where exactly). Calmav is calling function handle_stream to process the network payload (link). If the payload exceeds StreamMaxLength the handle_stream function:

1. calls conn_reply_error which sends INSTREAM size limit exceeded. to the socket (link)
2. set error to 1
3. return -1

After that the socket is closed (link)

Here is the debug information from calmd side when StreamMaxLength is exceeded:

WARNING: Thu Jul 28 12:32:11 2022 -> INSTREAM: Size limit reached, (requested: 104857600, max: 26214400)
Thu Jul 28 12:32:11 2022 -> Moved partial command: 4
Thu Jul 28 12:32:11 2022 -> mode == MODE_STREAM
Thu Jul 28 12:32:11 2022 -> Got chunksize: 104857600
Thu Jul 28 12:32:11 2022 -> Shutting down socket after error (FD 10)
Thu Jul 28 12:32:11 2022 -> Number of file descriptors polled: 1 fds

It looks like clamscan does not every time receive/process the information that the payload is too big and even if this information is properly received/processed clamscan does not properly handle such situation (it does not reject but resolve with NodeClamError)

[kylefarris]

@benzino77 If you have any spare time to look into how to resolve the issue of clamscan not receiving/processing the info consistently and submit a PR, I'd be very grateful. At the moment, I'm just a bit overloaded with a million other projects. Also, thank you so much for the investigative work you've done so far--it's been very helpful and informative.

[benzino77]

No problem. I will make PR with proposition.

[hanoj-budime]

I have investigated it a little bit more, and it looks like it can be a problem on clamscan side (don't know where exactly). Calmav is calling function handle_stream to process the network payload (link). If the payload exceeds StreamMaxLength the handle_stream function:

1. calls conn_reply_error which sends INSTREAM size limit exceeded. to the socket (link)
2. set error to 1
3. return -1

After that the socket is closed (link)

Here is the debug information from calmd side when StreamMaxLength is exceeded:

WARNING: Thu Jul 28 12:32:11 2022 -> INSTREAM: Size limit reached, (requested: 104857600, max: 26214400)
Thu Jul 28 12:32:11 2022 -> Moved partial command: 4
Thu Jul 28 12:32:11 2022 -> mode == MODE_STREAM
Thu Jul 28 12:32:11 2022 -> Got chunksize: 104857600
Thu Jul 28 12:32:11 2022 -> Shutting down socket after error (FD 10)
Thu Jul 28 12:32:11 2022 -> Number of file descriptors polled: 1 fds

It looks like clamscan does not every time receive/process the information that the payload is too big and even if this information is properly received/processed clamscan does not properly handle such situation (it does not reject but resolve with NodeClamError)

@benzino77 - Thanks for spending some time here - to help us! 😊

[hanoj-budime]

No problem. I will make PR with proposition.

Definitely, this approach going to help us. therefore we can notify such error in our services & then we'll throw based that to my HTTP client request

@benzino77 @kylefarris Once Again - Thank you guys ( For your Support ).

Looking forward to this PR 🏳

[benzino77]

Now I start thinking that it can be clamd problem. I've performed some tests on v0.103.6 clamd and it looks like clamscan is detecting socket error (write EPIPE), before any information is send back by clamd (that stream size limit is exceeded) almost every time (much, much more often than for v0.104.x or v0.105.x).

I'm starting my holidays so I will get back to this when I'll be back.

[kylefarris]

Thanks for the update @benzino77. Enjoy your holidays!