Reduce memory usage of pixel_to_pixel

[astrofrog]

This isn't really an issue that requires work in reproject but it might require work in several other packages. Basically pixel_to_pixel can be very memory-intensive under certain situations. Here's an example:

https://gist.github.com/astrofrog/a807da0c34b7d669289e3c14d43a099a

this is with a GWCS and regular astropy WCS object. The input arrays are 80Mb each (160Mb total) but the memory peaks around 1.7-1.8Gb which is 10x more than this. It might be there's not much we can do about it, but it'd be interesting to see if there are places where we memory could be used more efficiently. Obviously, the output arrays also need 80Mb each, but that's still a lot of overhead.

[Cadair]

I'm curious about how much of this is in the WCSes vs the coordinate transform graph.

[astrofrog]

There's a lot of modeling-related stuff in the flamegraph

[astrofrog]

The coordinate system might match in this case which would explain why not much coordinate-related stuff.

[Cadair]

oh good, looks like the next modeling task is memory optimisation lol