pyresample
pyresample copied to clipboard
Add aggregate/downsample, upsample, extend and reduce methods to AreaDefinition and SwathDefinition
This PR aims to provide the methods to aggregate/downsample, upsample, extend and reduce AreaDefinition and SwathDefinition objects. For SwathDefinition objects, it returns a new objects with the same inputs lats/lons type (numpy/dask/xr.DataArray).
- [ ] Closes
- [ ] #407
- [ ] #28
- [x] Tests added
- [x] Tests passed
- [ ] Passes
git diff origin/main **/*py | flake8 --diff - [x] Fully documented
Yet to be done/defined:
- [x] I added the method
downsamplemethod and added a deprecation warning toAreaDefinition.aggregate? - [x] Any reason to still use
pyproj.transforminstead of the more efficientpyproj.TransformerinSwathDefinition._do_transform? - [x] For
SwathDefinition.extend, do we define which pyproj.Geod?pyproj.Geod(ellps='sphere')orpyproj.Geod(ellps='WGS84')? - [ ] I tried to make
SwathDefinition.upsampledask-compatible. But I need some help/guidance to finish out the work. See between L828-L850 and L870-L880 ofgeometry.py. - [ ]
SwathDefinition.upsamplecould also use insteadgeotiepoints.GeoInterpolatorbut I am not sure I understood its possible correct usage for upsampling a swath grid. Additionally it seems not dask compatible. See below PR comment for a code attempt ... - [ ] Extend and reduce methods for GEO AreaDef are not implemented. They need ad-hoc code. Ideas?
PS: Is my first PR to a package, so please be patient and don't hesitate to provide a lot of feedback/reproaches :)
There are 238 errors:
- pyresample/geometry.py, line 718 - W291 trailing whitespace
- pyresample/geometry.py, line 719 - W293 blank line contains whitespace
- pyresample/geometry.py, line 720 - W291 trailing whitespace
- pyresample/geometry.py, line 721 - E501 line too long (124 > 120 characters)
- pyresample/geometry.py, line 727 - W291 trailing whitespace
- pyresample/geometry.py, line 732 - W293 blank line contains whitespace
- pyresample/geometry.py, line 733 - W291 trailing whitespace
- pyresample/geometry.py, line 735 - W291 trailing whitespace
- pyresample/geometry.py, line 740 - W291 trailing whitespace
- pyresample/geometry.py, line 743 - W293 blank line contains whitespace
- pyresample/geometry.py, line 744 - W291 trailing whitespace
- pyresample/geometry.py, line 747 - W291 trailing whitespace
- pyresample/geometry.py, line 749 - W293 blank line contains whitespace
- pyresample/geometry.py, line 750 - W291 trailing whitespace
- pyresample/geometry.py, line 751 - E225 missing whitespace around operator W291 trailing whitespace
- pyresample/geometry.py, line 752 - W291 trailing whitespace
- pyresample/geometry.py, line 753 - W293 blank line contains whitespace
- pyresample/geometry.py, line 754 - W291 trailing whitespace
- pyresample/geometry.py, line 757 - W293 blank line contains whitespace
- pyresample/geometry.py, line 759 - E231 missing whitespace after ','
- pyresample/geometry.py, line 760 - E231 missing whitespace after ','
- pyresample/geometry.py, line 761 - W293 blank line contains whitespace
- pyresample/geometry.py, line 764 - W291 trailing whitespace
- pyresample/geometry.py, line 766 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 770 - W293 blank line contains whitespace
- pyresample/geometry.py, line 771 - W291 trailing whitespace
- pyresample/geometry.py, line 773 - W293 blank line contains whitespace
- pyresample/geometry.py, line 774 - W291 trailing whitespace
- pyresample/geometry.py, line 776 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 777 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 778 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 781 - W293 blank line contains whitespace
- pyresample/geometry.py, line 783 - E231 missing whitespace after ','
- pyresample/geometry.py, line 784 - E231 missing whitespace after ','
- pyresample/geometry.py, line 785 - W293 blank line contains whitespace
- pyresample/geometry.py, line 788 - W291 trailing whitespace
- pyresample/geometry.py, line 790 - W291 trailing whitespace
- pyresample/geometry.py, line 793 - W291 trailing whitespace
- pyresample/geometry.py, line 798 - W293 blank line contains whitespace
- pyresample/geometry.py, line 799 - W291 trailing whitespace
- pyresample/geometry.py, line 801 - W293 blank line contains whitespace
- pyresample/geometry.py, line 804 - W293 blank line contains whitespace
- pyresample/geometry.py, line 806 - W291 trailing whitespace
- pyresample/geometry.py, line 816 - W293 blank line contains whitespace
- pyresample/geometry.py, line 819 - E231 missing whitespace after ','
- pyresample/geometry.py, line 820 - W293 blank line contains whitespace
- pyresample/geometry.py, line 823 - E231 missing whitespace after ',' W291 trailing whitespace
- pyresample/geometry.py, line 825 - W291 trailing whitespace
- pyresample/geometry.py, line 826 - E111 indentation is not a multiple of 4 E117 over-indented
- pyresample/geometry.py, line 827 - W291 trailing whitespace
- pyresample/geometry.py, line 829 - W291 trailing whitespace
- pyresample/geometry.py, line 831 - W291 trailing whitespace
- pyresample/geometry.py, line 833 - W293 blank line contains whitespace
- pyresample/geometry.py, line 838 - E225 missing whitespace around operator
- pyresample/geometry.py, line 840 - E231 missing whitespace after ','
- pyresample/geometry.py, line 842 - E261 at least two spaces before inline comment W291 trailing whitespace
- pyresample/geometry.py, line 844 - E231 missing whitespace after ','
- pyresample/geometry.py, line 846 - W293 blank line contains whitespace
- pyresample/geometry.py, line 851 - W293 blank line contains whitespace
- pyresample/geometry.py, line 856 - W293 blank line contains whitespace
- pyresample/geometry.py, line 860 - W293 blank line contains whitespace
- pyresample/geometry.py, line 861 - W291 trailing whitespace
- pyresample/geometry.py, line 862 - W291 trailing whitespace
- pyresample/geometry.py, line 873 - W293 blank line contains whitespace
- pyresample/geometry.py, line 879 - W293 blank line contains whitespace
- pyresample/geometry.py, line 880 - W291 trailing whitespace
- pyresample/geometry.py, line 883 - W293 blank line contains whitespace
- pyresample/geometry.py, line 885 - E231 missing whitespace after ','
- pyresample/geometry.py, line 886 - E231 missing whitespace after ','
- pyresample/geometry.py, line 887 - W293 blank line contains whitespace
- pyresample/geometry.py, line 890 - W291 trailing whitespace
- pyresample/geometry.py, line 892 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 896 - W293 blank line contains whitespace
- pyresample/geometry.py, line 897 - W291 trailing whitespace
- pyresample/geometry.py, line 900 - W291 trailing whitespace
- pyresample/geometry.py, line 902 - W291 trailing whitespace
- pyresample/geometry.py, line 905 - W293 blank line contains whitespace
- pyresample/geometry.py, line 906 - E231 missing whitespace after ',' W291 trailing whitespace
- pyresample/geometry.py, line 907 - E231 missing whitespace after ','
- pyresample/geometry.py, line 908 - E231 missing whitespace after ','
- pyresample/geometry.py, line 909 - W291 trailing whitespace
- pyresample/geometry.py, line 910 - W293 blank line contains whitespace
- pyresample/geometry.py, line 911 - W291 trailing whitespace
- pyresample/geometry.py, line 913 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 914 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 915 - E261 at least two spaces before inline comment
- pyresample/geometry.py, line 918 - W293 blank line contains whitespace
- pyresample/geometry.py, line 920 - E231 missing whitespace after ','
- pyresample/geometry.py, line 921 - E231 missing whitespace after ','
- pyresample/geometry.py, line 922 - W293 blank line contains whitespace
- pyresample/geometry.py, line 925 - W291 trailing whitespace
- pyresample/geometry.py, line 927 - W291 trailing whitespace
- pyresample/geometry.py, line 930 - W291 trailing whitespace
- pyresample/geometry.py, line 935 - W293 blank line contains whitespace
- pyresample/geometry.py, line 936 - W291 trailing whitespace
- pyresample/geometry.py, line 938 - W293 blank line contains whitespace
- pyresample/geometry.py, line 941 - W291 trailing whitespace
- pyresample/geometry.py, line 944 - W291 trailing whitespace
- pyresample/geometry.py, line 945 - W291 trailing whitespace
- pyresample/geometry.py, line 948 - W291 trailing whitespace
- pyresample/geometry.py, line 950 - W293 blank line contains whitespace
- pyresample/geometry.py, line 951 - W291 trailing whitespace
- pyresample/geometry.py, line 952 - E225 missing whitespace around operator W291 trailing whitespace
- pyresample/geometry.py, line 953 - W291 trailing whitespace
- pyresample/geometry.py, line 954 - W293 blank line contains whitespace
- pyresample/geometry.py, line 956 - E231 missing whitespace after ','
- pyresample/geometry.py, line 957 - E231 missing whitespace after ','
- pyresample/geometry.py, line 958 - W293 blank line contains whitespace
- pyresample/geometry.py, line 961 - W293 blank line contains whitespace
- pyresample/geometry.py, line 964 - E231 missing whitespace after ','
- pyresample/geometry.py, line 965 - E231 missing whitespace after ','
- pyresample/geometry.py, line 968 - E231 missing whitespace after ','
- pyresample/geometry.py, line 970 - E231 missing whitespace after ','
- pyresample/geometry.py, line 972 - W293 blank line contains whitespace
- pyresample/geometry.py, line 973 - W291 trailing whitespace
- pyresample/geometry.py, line 974 - W291 trailing whitespace
- pyresample/geometry.py, line 975 - E111 indentation is not a multiple of 4 E231 missing whitespace after ','
- pyresample/geometry.py, line 976 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 977 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 978 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 979 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 980 - E111 indentation is not a multiple of 4 E231 missing whitespace after ','
- pyresample/geometry.py, line 981 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 982 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 983 - W293 blank line contains whitespace
- pyresample/geometry.py, line 985 - E231 missing whitespace after ','
- pyresample/geometry.py, line 986 - E231 missing whitespace after ','
- pyresample/geometry.py, line 987 - W293 blank line contains whitespace
- pyresample/geometry.py, line 992 - W293 blank line contains whitespace
- pyresample/geometry.py, line 993 - W291 trailing whitespace
- pyresample/geometry.py, line 995 - W293 blank line contains whitespace
- pyresample/geometry.py, line 998 - W291 trailing whitespace
- pyresample/geometry.py, line 1006 - W291 trailing whitespace
- pyresample/geometry.py, line 1008 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1009 - E226 missing whitespace around arithmetic operator W291 trailing whitespace
- pyresample/geometry.py, line 1010 - E501 line too long (140 > 120 characters)
- pyresample/geometry.py, line 1011 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1012 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1013 - E501 line too long (141 > 120 characters)
- pyresample/geometry.py, line 1014 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1015 - W291 trailing whitespace
- pyresample/geometry.py, line 1016 - E225 missing whitespace around operator W291 trailing whitespace
- pyresample/geometry.py, line 1017 - W291 trailing whitespace
- pyresample/geometry.py, line 1018 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1020 - E226 missing whitespace around arithmetic operator E231 missing whitespace after ','
- pyresample/geometry.py, line 1021 - E226 missing whitespace around arithmetic operator E231 missing whitespace after ','
- pyresample/geometry.py, line 1700 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1706 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1707 - W291 trailing whitespace
- pyresample/geometry.py, line 1709 - W291 trailing whitespace
- pyresample/geometry.py, line 1711 - W291 trailing whitespace
- pyresample/geometry.py, line 1714 - W291 trailing whitespace
- pyresample/geometry.py, line 1716 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1718 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1719 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1725 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1726 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1727 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1728 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1729 - W291 trailing whitespace
- pyresample/geometry.py, line 1730 - W291 trailing whitespace
- pyresample/geometry.py, line 1732 - W291 trailing whitespace
- pyresample/geometry.py, line 1735 - W291 trailing whitespace
- pyresample/geometry.py, line 1737 - W291 trailing whitespace
- pyresample/geometry.py, line 1739 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1741 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1744 - W291 trailing whitespace
- pyresample/geometry.py, line 1748 - W291 trailing whitespace
- pyresample/geometry.py, line 1751 - W291 trailing whitespace
- pyresample/geometry.py, line 1753 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1754 - E226 missing whitespace around arithmetic operator W291 trailing whitespace
- pyresample/geometry.py, line 1756 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1757 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1759 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1760 - W291 trailing whitespace
- pyresample/geometry.py, line 1761 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1762 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1768 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1769 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1770 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1771 - E226 missing whitespace around arithmetic operator
- pyresample/geometry.py, line 1772 - W291 trailing whitespace
- pyresample/geometry.py, line 1773 - W291 trailing whitespace
- pyresample/geometry.py, line 1775 - W291 trailing whitespace
- pyresample/geometry.py, line 1778 - W291 trailing whitespace
- pyresample/geometry.py, line 1780 - W291 trailing whitespace
- pyresample/geometry.py, line 1782 - W293 blank line contains whitespace
- pyresample/geometry.py, line 1784 - W293 blank line contains whitespace
- pyresample/geometry.py, line 3128 - E302 expected 2 blank lines, found 1 W291 trailing whitespace
- pyresample/geometry.py, line 3130 - W291 trailing whitespace
- pyresample/geometry.py, line 3141 - W291 trailing whitespace
- pyresample/geometry.py, line 3150 - W291 trailing whitespace
- pyresample/geometry.py, line 3157 - W291 trailing whitespace
- pyresample/geometry.py, line 3158 - E231 missing whitespace after ','
- pyresample/geometry.py, line 3159 - W291 trailing whitespace
- pyresample/geometry.py, line 3165 - W291 trailing whitespace
- pyresample/geometry.py, line 3166 - W291 trailing whitespace
- pyresample/geometry.py, line 3172 - W291 trailing whitespace
- pyresample/geometry.py, line 3174 - W291 trailing whitespace
- pyresample/geometry.py, line 3178 - W291 trailing whitespace
- pyresample/geometry.py, line 3179 - W291 trailing whitespace
- pyresample/geometry.py, line 3181 - W291 trailing whitespace
- pyresample/geometry.py, line 3185 - E203 whitespace before ',' W291 trailing whitespace
- pyresample/geometry.py, line 3187 - W291 trailing whitespace
- pyresample/geometry.py, line 3190 - W291 trailing whitespace
- pyresample/geometry.py, line 3191 - W293 blank line contains whitespace
- pyresample/geometry.py, line 3192 - W291 trailing whitespace
- pyresample/geometry.py, line 3193 - W291 trailing whitespace
- pyresample/geometry.py, line 3197 - W291 trailing whitespace
- pyresample/geometry.py, line 3201 - W291 trailing whitespace
- pyresample/geometry.py, line 3204 - W291 trailing whitespace
- pyresample/geometry.py, line 3205 - W293 blank line contains whitespace
- pyresample/geometry.py, line 3206 - W291 trailing whitespace
- pyresample/geometry.py, line 3207 - W291 trailing whitespace
- pyresample/geometry.py, line 3218 - W291 trailing whitespace
- pyresample/geometry.py, line 3219 - W293 blank line contains whitespace
- pyresample/geometry.py, line 3220 - W291 trailing whitespace
- pyresample/geometry.py, line 3223 - E302 expected 2 blank lines, found 1
- pyresample/geometry.py, line 3224 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3225 - W291 trailing whitespace
- pyresample/geometry.py, line 3226 - W291 trailing whitespace
- pyresample/geometry.py, line 3228 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3229 - E111 indentation is not a multiple of 4 E261 at least two spaces before inline comment
- pyresample/geometry.py, line 3230 - E114 indentation is not a multiple of 4 (comment)
- pyresample/geometry.py, line 3231 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3232 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3233 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3234 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3235 - E111 indentation is not a multiple of 4 W291 trailing whitespace
- pyresample/geometry.py, line 3237 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3238 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3239 - W293 blank line contains whitespace
- pyresample/geometry.py, line 3240 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3241 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3242 - E111 indentation is not a multiple of 4 W291 trailing whitespace
- pyresample/geometry.py, line 3243 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3244 - E111 indentation is not a multiple of 4
- pyresample/geometry.py, line 3245 - E111 indentation is not a multiple of 4
pre-commit.ci autofix
Codecov Report
Merging #413 (2e5997c) into main (565966f) will increase coverage by
0.31%. The diff coverage is99.83%.
:exclamation: Current head 2e5997c differs from pull request most recent head e80f44e. Consider uploading reports for the commit e80f44e to get more accurate results
@@ Coverage Diff @@
## main #413 +/- ##
==========================================
+ Coverage 93.86% 94.18% +0.31%
==========================================
Files 65 65
Lines 11125 11706 +581
==========================================
+ Hits 10443 11025 +582
+ Misses 682 681 -1
| Flag | Coverage Δ | |
|---|---|---|
| unittests | 94.18% <99.83%> (+0.31%) |
:arrow_up: |
Flags with carried forward coverage won't be shown. Click here to find out more.
| Impacted Files | Coverage Δ | |
|---|---|---|
| pyresample/geometry.py | 89.44% <99.66%> (+2.43%) |
:arrow_up: |
| pyresample/test/test_geometry.py | 99.43% <100.00%> (+0.12%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 565966f...e80f44e. Read the comment docs.
Coverage increased (+0.3%) to 94.012% when pulling 2e5997c9d7a14111866c0f4c9eca71bc6ba92af6 on ghiggi:feature-upsample-downsample into 565966f210f6ae4c2d1d2ba32a557f4a13ec505f on pytroll:main.
This paragraph is to discuss the possibility of using geotiepoints.GeoInterpolator within SwathDefinition.upsample.
I might have misunderstood the usage of the GeoInterpolator, but I can't retrieve the correct upsampled centroids:
import numpy as np
from geotiepoints.geointerpolator import GeoInterpolator
# Data Example 1
x = 2
y = 2
src_lons = np.array([[5,9],
[5,9],
[5,9],
[5,9]])
src_lats = np.array([[6,6],
[4,4],
[0,0],
[-2,-2]])
# Data Example 2
lons = np.arange(-179.5, 179.5 + 0.5, 0.5)
lats = np.arange(-89.5, 0.5, 0.5)
src_lons, src_lats = np.meshgrid(lons, lats)
x = 2
y = 2
#--------------------------------
# Workflow
height = src_lons.shape[0]
width = src_lons.shape[1]
tie_cols = np.arange(0, width*2, 2)
fine_cols = np.arange(-1 + 1/x, width*2-1, 1/x*2)
fine_rows = np.arange(-1 + 1/y, height*2-1, 1/y*2)
tie_rows = np.arange(0, height*2, 2)
# Print ties
print(fine_rows)
print(tie_rows)
print(fine_cols)
print(tie_cols)
#--------------------------------
interpolator = GeoInterpolator((src_lons, src_lats),
(tie_rows, tie_cols),
(fine_rows, fine_cols),
2, 2)
interpolator.fill_borders("y", "x")
lons_up, lats_up = interpolator.interpolate()
print("With fill borders:")
print(np.round(lats_up[:,0],2)) # stop at lats 0.5 instead of -1.5 (ex1) and -8.78 instead of 0.5 (ex2)
interpolator = GeoInterpolator((src_lons, src_lats),
(tie_rows, tie_cols),
(fine_rows, fine_cols),
1, 1) # 2,2, does not work with ex1
lons_up, lats_up = interpolator.interpolate()
print("")
print("Without fill borders:")
print(np.round(lats_up[:,0],2)) # With ex2, not using fill_borders enable to reach at least lat 0, but do not estimate the borders
The code below documents an uncommon SwathDefinition edge case where swath_def.upsample(2,2).aggregate(2,2) does not return the same source swath_def (only close to the poles). It is an unrealistic swath, since everything along y direction points toward the pole ...
import numpy as np
from pyresample.geometry import SwathDefinition
np.set_printoptions(suppress=True)
lons = np.arange(-179.75, 179.75 + 0.5, 0.5)
lats = np.arange(-89.75, 0.5, 0.5)
lons, lats = np.meshgrid(lons, lats)
swath_def = SwathDefinition(lons=lons, lats=lats)
### Precision error (?) between upsampling and aggregating
swath_def_rev = swath_def.upsample(y=2, x=2).aggregate(x=2, y=2)
np.allclose(swath_def_rev.lons, swath_def.lons, atol=1e-01)
np.allclose(swath_def_rev.lats, swath_def.lats, atol=1e-03)
np.isclose(swath_def_rev.lons, swath_def.lons, atol=1e-08)
bool_arr = np.isclose(swath_def_rev.lons, swath_def.lons, atol=1e-08)
swath_def.lats[~bool_arr.any(axis=1),0] # problematic latitudes
np.sum(bool_arr)/np.size(bool_arr)*100
Also, this review took me more than 2 hours, in the future please consider making more incremental changes in each PR :sweat_smile:
I plan to do the changes tomorrow afternoon @djhoese @mraspaud . Should I resolve the conservation comments after having resolved your concerns?
Better leave them open if it's not trivial, so we can have another look after it's fixed and approve.