gfx_RotateScaleSprite breaks when scale >= 128
both gfx_RotateScaleSprite and gfx_RotatedScaledSprite_NoClip break when scale is >= 128 (>= 200%). The documentation says that range is about 1% to 400% scale or 1 to 255.
When the scale is >= 128, the sprite is rotated in the opposite direction + 180 degrees, and will start decreasing in size.
Shown below is the scale sweeping from 4 to 255:
I've been able to fix the rotation when scale is >= 128. But fixing the scaling for a scale of [129, 255] is more complicated as it is probably an 8bit/16bit overflow/underflow.
With larger sprites, a different bug occurs when sprite_width * scale exceeds 16384, preventing (sprite_width * sprite_height / 64) from fitting inside 8 bits. This makes it so only the top-left corner of the sprite is sampled.
This bug doesn't occur for the example shown above, since 48 * 255 is less than 16384.
Everything has been fixed except for the case that the output is 256x256 or larger. This is not worth fixing however, since a 256x256 sprite doesn't fit onto a 320x240 screen anyways, and would require changing 8 bit counters to be 9 or 16 bits.