respmod_copy.py errors on files > 65535 bytes

[cyberxml]

setup a squid+pyicap server using respmod_copy.py from the examples directory

on an ftp or http server, prepare two files head -c 65535 < /dev/urandom > file-65535.txt head -c 65536 < /dev/urandom > file-65536.txt

on the ftp client, attempt to retrieve each file curl -m 10 -x squidproxy:3128 ftp://username:userpass@netserver:21/file-65535.txt (pass) curl -m 10 -x squidproxy:3128 ftp://username:userpass@netserver:21/file-65536.txt (fail) curl -m 10 -x squidproxy:3128 http://netserver/file-65535.txt (pass) curl -m 10 -x squidproxy:3128 http://netserver/file-65536.txt (fail)

Error generated is

Exception happened during processing of request from ('127.0.0.1', 50874) Traceback (most recent call last) File "/usr/lib64/python2.7/SocketServer.py", line 593, in process_request_thread self.finish_request(request, client_address) File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.7/SocketServer.py", line 649, in init self.handle() File "/usr/lib/python2.7/site-packages/pyicap.py", line 443, in handle self.handle_one_request() File "/usr/lib/python2.7/site-packages/pyicap.py", line 481, in handle_one_request self.raw_requestline = self.rfile.readline(65537) File "/usr/lib64/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) error: [Errno 104] Connection reset by peer

[cyberxml]

icap portion of squid.conf (recall this works for all files < 65536 bytes)

#icap config for pyicap icap_enable on icap_preview_enable on icap_preview_size 1024 icap_service service_resp respmod_precache bypass=1 icap://127.0.0.1:1344/example adaptation_access service_resp allow all

[cyberxml]

Looking at the tcpdump trace for the loopback (respmod_copy pyicap service is on same server as squid), the full file is being returned to squid in both instances. When the final continuation for the 65535 byte file is transferred from ICAP to SQUID, squid replies with FIN,ACK. When the final continuation for the 65536 byte file is transferred from ICAP to SQUID, squid replies with a RST, ACK.

[cyberxml]

Examining the pcaps more closely, I see that an ICAP header with "Allow: 204" is sent in the case of a file with 65535 bytes but not sent when the file is 65536 or more. So this might be a squid thing. Can override in OPTIONS?

[cyberxml]

This may be relevant

/My icap conf activated preview and preview size as follow : />/icap_preview_enable on />/icap_preview_size 1024 / IIRC, Squid ignores icap_preview_size in squid.conf (a bug). The ICAP service OPTIONS response determines the Preview size (subject to an internal limit of 64KB).

http://squid-web-proxy-cache.1019090.n4.nabble.com/ICAP-and-Allow-204-Header-td4675762.html

[cyberxml]

Confirmed the 64KB limit is due to Squid handling of Preview. Updated limit in squid source code, compiled, and retested. Larger files could transfer with the modified code.

But how can you pass files outside of the Preview feature? For example squid+squidclamav+c_icap passes large files for ClamAV scanning.