Enforce `Vararg` specialization

[MilesCranmer]

I didn't realize this but apparently Julia doesn't actually specialize functions based on variable numbers of arguments.

There's a way to get around this as described here: https://docs.julialang.org/en/v1/manual/performance-tips/#Be-aware-of-when-Julia-avoids-specializing.

This PR implements this forced specialization which should hopefully give some speedups.