Handle invalid windowBits in init functions

[stoeckmann]

Negative windowBits arguments are eventually turned positive in deflateInit2_ and inflateInit2_ (more precisely in inflateReset2). Such values are used to indicate that raw deflate/inflate should be performed.

If a user supplies INT_MIN for windowBits, the code will perform -INT_MIN which does not fit into int. In fact, this is undefined behavior in C and should be avoided.

Clearly this is a user error, but given the careful validation of input arguments a few lines later in deflateInit2_ I think this might be of interest.

Proof of Concept:

• Compile zlib with gcc -ftrapv or -fsanitize=undefined
• Compile and run this program:

 #include <limits.h>
 #include <stdio.h>
 #include <zlib.h>

 int main(void) {
	z_stream de_stream = { 0 }, in_stream = { 0 };
	int result;

	result = deflateInit2(&de_stream, 0, Z_DEFLATED, INT_MIN,
	    MAX_MEM_LEVEL, Z_DEFAULT_STRATEGY);
	printf("deflateInit2: %d\n", result);

	result = inflateInit2(&in_stream, INT_MIN);
	printf("inflateInit2: %d\n", result);

	return 0;
 }