satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Round satellite altitude in cache key to nearest 1000 for better cache hits

Open Plantain opened this issue 3 years ago • 6 comments

Himawari-8's altitude varies by +/- 500m from day to day, which means the hit rate on the cache for AHI sensor angles is so low that caching is not useful.

Maybe +/- 500m is different enough that this shouldn't be applied, I don't know.

Plantain avatar Mar 02 '22 07:03 Plantain

Codecov Report

Merging #2042 (bafe54e) into main (bafe54e) will not change coverage. The diff coverage is n/a.

:exclamation: Current head bafe54e differs from pull request most recent head 2b394d9. Consider uploading reports for the commit 2b394d9 to get more accurate results

@@           Coverage Diff           @@
##             main    #2042   +/-   ##
=======================================
  Coverage   93.88%   93.88%           
=======================================
  Files         283      283           
  Lines       43093    43093           
=======================================
  Hits        40458    40458           
  Misses       2635     2635           
Flag Coverage Δ
behaviourtests 4.78% <0.00%> (ø)
unittests 94.53% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update bafe54e...2b394d9. Read the comment docs.

codecov[bot] avatar Mar 02 '22 07:03 codecov[bot]

Coverage Status

Coverage increased (+0.0003%) to 94.145% when pulling 4e5c560a1252154785a9fd2b47b268af6b53ec2d on Plantain:main into 346e32dcd0c241be26635cc2cdd4b5d55fabe276 on pytroll:main.

coveralls avatar Mar 02 '22 08:03 coveralls

Where are we on this PR?

mraspaud avatar Apr 13 '22 13:04 mraspaud

I think we are waiting on confirmation that the AHI HSD reader's rounding isn't satisfactory in an operational sense. That means confirming that the satellite actual altitude and satellite actual longitude and latitude (not nadir) are being used in the angle generation and seeing what cache keys result (or would result) from a day or two of real AHI HSD data.

I can try checking parts of this later today, but either way I don't think this PR should be merged as-is without a lot more discussion.

djhoese avatar Apr 13 '22 13:04 djhoese

Ok I did a very small amount of investigating. There are multiple issues, but one big one is that rounding per-segment doesn't handle the merging of the metadata for all the segments. So in one of my test cases from 2018-11-12 18:30Z the rounded altitude (by 150m) result in two values: 35787000 and 35786850. When the combine_metadata handling averages these (np.mean in satpy/readers/file_handlers.py:_combine_orbital_parameters) the final result is 35786955. This isn't exactly intended because I wanted the final cached result to be the nice "clean" 150m rounded value.

Additionally the lon and lat for a dataset at 23:30 on this same day are 0.005 degrees and 0.05 degrees different (respectively) from this case. So both are rounded to different values...on the same day. Not ideal.

djhoese avatar Apr 15 '22 14:04 djhoese

I've been looking through a timeseries of Himawari position, in 2.5 years the satellite moved within: A latitude range of -0.5 -> +0.5 degrees (1.0 deg) A longitude range of 140.61 -> 140.71 degrees (0.1 deg) An altitude range of 35778.3 -> 35793.7km (15.4km).

Might help visualise what range of values the cache will expect to see!

simonrp84 avatar May 09 '22 12:05 simonrp84