ARM compilation doesn't like varargs hack in xdr.h

[matvore]

In xdr.h, there is a typedef:

typedef bool(*xdrproc_t) (XDR *, ...);

Which is invoked sometimes with 2, sometimes 3 arguments. The implementor of this function is not expected to write an actual varargs function, which is why I think of this as a hack. It is relying on implementation-specific behavior in C. When building for x86_64, this hack works. For Apple ARM, this does not. The second argument is never passed correctly and this causes a segment violation when it is dereferenced.

I did some experiments on both ARM and AMD64 [1] to see what the options are to fix. Based on this, I think that just changing the typedef to match the #ifdef _KERNEL branch [0] and updating call sites to pass a dummy value for the third argument is fine.

Another option, perhaps safer looking, though verbose, is to use a union of two function pointers, where one has 2 args and one has 3. Then the call site decides which one to use.

[0] typedef bool(*xdrproc_t) (XDR *, ...); [1] This also demonstrates that varargs works for AMD64 but not for ARM64:

/* foo.c */
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

typedef void (*w)(void *x, ...);
typedef void (*x)(void *x, void *y);
typedef void (*y)(void *x, void *y, unsigned);

int rnd_fd;

void *rnd_ptr(void)
{
	void *r;

	if (sizeof(r) != read(rnd_fd, &r, sizeof(r))) err(1, "read");

	return r;
}

void func2(void *x, void *y)
{
	printf("%p %p\n", x, y);
}

void func3(void *x, char *y, unsigned z)
{
	printf("%p %p %x\n", x, y, z);
}

void invoke_w2(w fn)
{
	void *a = rnd_ptr(), *b = rnd_ptr();
	printf("%p %p\n", a, b);
	fn(a, b);
}

void invoke_w3(w fn)
{
	void *a = rnd_ptr(), *b = rnd_ptr(), *c = rnd_ptr();
	printf("%p %p %p\n", a, b, c);
	fn(a, b, c);
}

void invoke_x2(x fn)
{
	void *a = rnd_ptr(), *b = rnd_ptr();
	printf("%p %p\n", a, b);
	fn(a, b);
}

void invoke_y3(y fn)
{
	void *a = rnd_ptr(), *b = rnd_ptr(), *c = rnd_ptr();
	printf("%p %p %p\n", a, b, c);
	fn(a, b, c);
}

int main(int argc, char **argv)
{
	system("arch");

	rnd_fd = open("/dev/random", O_RDONLY);

	if (rnd_fd < 0) err(1, "open");

	puts("Use correct number of arguments, but mismatching pointer types");
	invoke_x2(func2);

	puts("Correct number of args with mismatching pointer, and wrong int size");
	invoke_y3(func3);

	puts("Pass too many arguments");
	invoke_y3(func2);

	puts("Uses varargs");
	invoke_w2(func2);
	invoke_w3(func3);

	close(rnd_fd);
}

sample output:

$ rm foo; arch -x86_64 sh -c 'make foo && ./foo'; rm foo; arch -arm64 sh -c 'make foo && ./foo'
...
i386
Use correct number of arguments, but mismatching pointer types
0x1ac5a56d26e5f3f2 0x7ec5b08b39692a23
0x1ac5a56d26e5f3f2 0x7ec5b08b39692a23
Correct number of args with mismatching pointer, and wrong int size
0xadcf87966a70c79f 0x97980b5be36436f4 0x89fc5ca997f654a3
0xadcf87966a70c79f 0x97980b5be36436f4 97f654a3
Pass too many arguments
0x1a0b7cbbc56352c2 0x38e49162dc53e142 0xf561f669cab2a714
0x1a0b7cbbc56352c2 0x38e49162dc53e142
Uses varargs
0xa96d0453cd286a1b 0xc8aff6e46fd4e6ed
0xa96d0453cd286a1b 0xc8aff6e46fd4e6ed
0x9f7dabc6a56ea0b1 0x809b334096f91f4a 0x45441ca9278e90ba
0x9f7dabc6a56ea0b1 0x809b334096f91f4a 278e90ba
...
arm64
Use correct number of arguments, but mismatching pointer types
0x457c6b92b58d7940 0xbd05049dcb4b515b
0x457c6b92b58d7940 0xbd05049dcb4b515b
Correct number of args with mismatching pointer, and wrong int size
0x2cdba0028184be72 0x3eaee9ed190385db 0x5e050c6d15c0dc27
0x2cdba0028184be72 0x3eaee9ed190385db 15c0dc27
Pass too many arguments
0x94e759310ccae495 0xafac8c62267e21c6 0x5048af81d7b7c93a
0x94e759310ccae495 0xafac8c62267e21c6
Uses varargs
0xda0263b66698276a 0x251539b9633de50f
0xda0263b66698276a 0x0
0x6f160ab22f897aa2 0xc147c48bf64cdba8 0xde3610eb3e19e7
0x6f160ab22f897aa2 0x0 120a8

[matvore]

This problem is mentioned in the version of rpc/xdr.h that is bundled with Xcode - available here: https://opensource.apple.com/source/Libinfo/Libinfo-517.30.1/rpc.subproj/xdr.h

According to the below, casting a 2-arg function ptr to a 3-arg one is safe in at least this use case (where the arg types are ptr, ptr, int).

This is how the typedef is documented and defined:

/*
 * A xdrproc_t exists for each data type which is to be encoded or decoded.
 *
 * The second argument to the xdrproc_t is a pointer to an opaque pointer.
 * The opaque pointer generally points to a structure of the data type
 * to be decoded.  If this pointer is 0, then the type routines should
 * allocate dynamic storage of the appropriate size and return it.
 *
 *
 * IMPORTANT NOTE: Apple iOS and OS X
 * 
 * Previous versions of this header file defined the xdrproc_t prototype as 
 *     typedef bool_t (*xdrproc_t)(XDR *, ...); 
 * 
 * This prototype is incorrect.  One may not use a varargs function pointer
 * in place of pointer to a function with positional parameters.
 * This mistake has been masked by the fact that clients of this API would
 * generally cast their xdr functions as (xdrproc_t), and thus silence compiler
 * warnings.  The code worked because the ABI for varargs routines that pass a
 * small number of arguments has been the same as the ABI for routines with a
 * few positional parameters.  However, if the ABI differs this will cause the
 * compiled code to fail.
 *
 * Historically, some client xdr routines expected three parameters.  This 
 * does not seem to be the case in more recent code, but we have decided to
 * retain this definition in the XDR library in case some legacy code still
 * expects three parameters.  The library will pass zero as the third
 * parameter.  Routines that expect two parameters will work correctly.
 *
 * If your client-side xdr routine is of the form:
 *   bool_t xdr_my_datatype(XDR *, void *);
 * and you pass your function pointer to routines like xdr_pointer or
 * xdr_reference using "(xdrproc_t)xdr_my_datatype", then your code will
 * compile and run correctly.  If your code invokes an xdrproc_t callback,
 * it must be modified to pass a third parameter, which may simply be zero.
 */
typedef	bool_t (*xdrproc_t)(XDR *, void *, unsigned int);

[ffilz]

Looks like this was fixed.