ETags adjusted for "content-encoding" are not correct

[kanongil]

Support plan

• is this issue currently blocking your project? (yes/no): no
• is this issue affecting a production system? (yes/no): probably not

Context

• node version: any
• module version with issue: 20.0.3
• last module version without issue: none
• environment (e.g. node, browser, native): node
• used with (e.g. hapi application, another framework, standalone, ...): standalone
• any other relevant information:

What are you trying to achieve or the steps to reproduce?

Serve wellformed responses that won't corrupt the received payload.

const server = Hapi.server({ port: 4000 });
server.route({
    method: 'get',
    path: '/',
    handler(request, h) {

        const response = h.entity({ etag: 'abc' });
        if (response) {
            return response;
        }

        const total = 4000;
        const bytes = parseInt(request.query.i) || 100;
        const stream = new Stream.Readable({ read() {} });
        stream.setCompressor = (compressor) => {

            setImmediate(() => {

                let sent = 0;
                for (sent = 0; sent < total - bytes; sent += bytes) {
                    stream.push(Buffer.alloc(bytes));
                    compressor.flush();
                }

                stream.push(Buffer.alloc(total - sent));
                stream.push(null);
            });
        };

        return h.response(stream).type('text/html');
    }
});

server.start();

A test:

curl -H 'Accept-Encoding: gzip' 'localhost:4000/?i=500' -v | wc -c

ETag: "abc-gzip" Bytes: 110

When the same content is requested with different flushing the results are wrong:

curl -H 'Accept-Encoding: gzip' 'localhost:4000/?i=20' -v | wc -c

What was the result you got?

ETag: "abc-gzip" Bytes: 2013

What result did you expect?

I would expect both results to signal a weak ETag since the transferred bytes are not always identical. In this case that would be:

ETag: W/"abs"

While I expect most compressed responses are bytewise identical, this is not a guarantee and hapi should not expect this as the default.

[devinivy]

Would your recommendation be to always default to weak etags when serving compressed files rather than encoding-suffixed strong etags? Or perhaps only do this when the response stream has a custom setCompressor()? I wonder if there also might be an opportunity for h.entity() to support the same weak option provided by response.etag().

[kanongil]

Yes, just always set a weak etag instead of the -gzip (when adding a compressor). I don't believe this will have any adverse effect.

The fact that I can provoke the issue using flushing is just a detail. The issue can in theory occur on any normal transmission, as there is no requirement for a compressor to produce the same output from the same input. Another way to provoke the issue, is to set a custom compressor and change the compression level (eg. in response to server CPU load).

[kanongil]

Afaik, a browser or caching proxy only needs strong ETags when it has a partial response and wants to expand on this using a Range request. For normal caching purposes weak ETags are fine.