The reproject performance

[fockez]

I am trying to reproject to stack about 20 4k x 4k frames, and find that it takes too much time. After I check the cProfile result, I found that most time spend procedure are related wcs convertion, is that correct? And more, I use SIP wcs, the astropy deal with SIP using original python code not wcslib code, is that related? Thanks.

[astrofrog]

@fockez - could you provide the cProfile output here? How long are things taking currently? If you can show the same performance issues by just parsing the WCS with astropy and showing that e.g wcs.all_pix2world and wcs.all_world2pix are much slower than wcs.all_pix2world and wcs.all_world2pix (which ignores distortions) it might be worth opening an issue on the core astropy repository?

[fockez]

1.zip I just do a single call of reproject_interp for test on 4k image and attatch the cProfile log. I wonder if there is optimize space. Thanks. Also, I test the distortions, it doubles the time comsuption in wcs.py. 2.zip This is no distortion version.

[fockez]

Because the wcs I use doesn't have SIP AP and BP coeffcient, so the world2pix is obviously slow than pix2world. And more, the round trip pixel to pixel cost two times. I see that there is no round trip before, so is that optional just for check?

[astrofrog]

@fockez - if the time is just doubled because of the distortions, this doesn't seem too bad a performance penalty. We indeed now do round-tripping in the reprojection because there are some cases where we need to check that we are not including positions that are on the other side of the sphere where the data is defined (this can happen with solar data). There are two things we could consider doing here:

• Investigating whether the distortion transformation code in astropy could be re-written in Cython or C to speed it up. This is a reasonably big chunk of work, and I'm not sure if anyone would have the time to do this at the moment.

• Add a flag to reproject to optionally disable the round-tripping check. This would definitely be easier and would save you a factor of ~2 in performance. Are you able to try turning off the round-tripping check to see if it affects your results?

[fockez]

I test to disable the round-trip check, the consumption lower from 31.7s to 17.1s. It is a big step. If this only affect in some cases, I think it is a good idea to have a switch to turn off the check, the default value can be true. About the wcs part, I think using wcslib SIP procedure may help, but I am not sure without test. I will add AP and BP coffcients to my wcs to see if this helps. I believe this will accelerate the world2pix, and this is the advantage of SIP, no need to iterate. Furthermore, optimizing the wcs procedure will be better but I don't know if there is the possibility. At first I though the projection will be the bottleneck, but in fact is the wcs part, beyound my hunch.

[fockez]

I added inverse sip coeffcients ap bp to the header, and found that the time is the same. After digging in, it shows that reproject use the wcs.all_world2pix function which does not consider the AP and BP, always use the iteration method. Unless I split this function into wcs_world2pix and sip_foc2pix.

[entaylor]

I'm also very interested in seeing this improved, so commenting here both for visibility and to subscribe to the thread.

[svank]

With #275 there is now a flag to disable roundtrip checking for the interpolating and adaptive algorithms. That PR has been merged, so the flag should appear in the next release.

[entaylor]

Thanks for this, @svank. Apologies for my ignorance, but can you give a rough sense for when the next release might be? If the next official release is some months off, should i just have a go pulling main from here (with all appropriate caveats)?

[svank]

I'm not a maintainer here, but I suspect the next release will just be whenever it's ready and the right people have some time. If you do pull the current main branch, afaik it's in a consistent state and should be useful for playing around, but I'd probably hesitate to do anything final or published without a proper release.

[Cadair]

I think we would like to get at least all of @svank's current chain of PRs merged before the next release, so it wont be imminent, but hopefully soon.