arcgis-python-api
arcgis-python-api copied to clipboard
cloud_removal_slra
Checklist
Please go through each entry in the below checklist and mark an 'X' if that condition has been met. Every entry should be marked with an 'X' to be get the Pull Request approved.
- [ ] All
import
s are in the first cell?- [ ] First block of imports are standard libraries
- [ ] Second block are 3rd party libraries
- [ ] Third block are all
arcgis
imports? Note that in some cases, for samples, it is a good idea to keep the imports next to where they are used, particularly for uncommonly used features that we want to highlight.
- [ ] All
GIS
object instantiations are one of the following?-
gis = GIS()
-
gis = GIS('home')
orgis = GIS('pro')
-
gis = GIS(profile="your_online_portal")
-
gis = GIS(profile="your_enterprise_portal")
-
- [ ] If this notebook requires setup or teardown, did you add the appropriate code to
./misc/setup.py
and/or./misc/teardown.py
? - [ ] If this notebook references any portal items that need to be staged on AGOL/Python API playground, did you coordinate with a Python API team member to stage the item the correct way with the
api_data_owner
user? - [ ] If the notebook requires working with local data (such as CSV, FGDB, SHP, Raster files), upload the files as items to the Geosaurus Online Org using
api_data_owner
account and change the notebook to first download and unpack the files. - [ ] Code simplified & split out across multiple cells, useful comments?
- [ ] Consistent voice/tense/narrative style? Thoroughly checked for typos?
- [ ] All images used like
<img src="base64str_here">
instead of<img src="https://some.url">
? All map widgets contain a static image preview? (Callmapview_inst.take_screenshot()
to do so) - [ ] All file paths are constructed in an OS-agnostic fashion with
os.path.join()
? (Instead ofr"\foo\bar"
,os.path.join(os.path.sep, "foo", "bar")
, etc.) - [x] Is your code formatted using Jupyter Black? You can use Jupyter Black to format your code in the notebook.
- [x] IF YOU WANT THIS SAMPLE TO BE DISPLAYED ON THE DEVELOPERS.ARCGIS.COM WEBSITE, ping @ mohi9282 so he can add it to the list for the next deploy
Check out this pull request on
See visual diffs & provide feedback on Jupyter Notebooks.
Powered by ReviewNB
Hi @sdash77 @anushriyaj
- I built this branch locally to preview - it looks great but just needs @anushriyaj to review for technical content
- The title text might need adjusted for capitalization:
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-27T04:36:11Z ----------------------------------------------------------------
Change the notebook file name to : cloud-removal-using-deep-learning-and-raster-function
Change notebook heading to: Cloud removal using pretrained deep learning model and raster function
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-27T04:36:12Z ----------------------------------------------------------------
please note that passwords are not expected to be exposed
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-27T04:36:12Z ----------------------------------------------------------------
raster and layer package must be published on python api playground portal using api data owner account. FYI deldevs is not an enterprise portal. It's an Online org.
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-27T04:36:13Z ----------------------------------------------------------------
do not keep hyphen in the word pretrained
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:55Z ----------------------------------------------------------------
Clouds often create obstructions in satellite imagery that can hinder results in processes like land cover classification and time series analysis. In this notebook, we will detect and replace cloud contaminated regions of satellite imagery by combining a pre-trained deep learning model, available in living atlas, and an image processing technique, available in raster functions.
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:55Z ----------------------------------------------------------------
Use the pre-trained cloud mask generation model
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:56Z ----------------------------------------------------------------
The pre-trained cloud mask generation model has been trained on sentinel-L2A imagery and is used to generate a cloud mask for an input raster. The cloud mask will be used by a Stepwise local radiometric adjustment (SLRA) raster function as guidance for removing cloud contaminated pixels from the target raster.
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:57Z ----------------------------------------------------------------
Reclassify cloud types in the cloud mask
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:58Z ----------------------------------------------------------------
The pre-trained cloud mask model will classify clouds into 3 classes. low, medium, and high density. As we want to remove all cloud contaminated regions, we will reclassify these 3 classes into one single class.
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:51:59Z ----------------------------------------------------------------
Use the stepwise local radiometric adjustment raster function
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:52:00Z ----------------------------------------------------------------
Next, we will use the Python raster function to blend areas using a Stepwise Local Radiometric Adjustment algorithm (Li .et al, 2019). Stepwise Local Radiometric Adjustment is used to fill contaminated areas and is performed on each mask region of the target image. Using the replacement raster, the tool will replace pixels that are covered with clouds.
The raster functions can be downloaded as a python(.py) file here in the functions folder. StepwiseLocalRadiometricAdjustment.py can be opened using Open Python Raster function
. Once opened, save it using Save As
. Size of Window
and Buffer Mask
can be experimented with values for varying results according to requirements and cloud coverage. Next, you can experiment with different values for the Size of Window and Buffer Mask parameters, depending on the requirements and cloud coverage of your use case.
Input Raster
: cloud covered imageryInput Replacement raster
: replacement rasterInput Mask
: generated cloud maskSize of Window
: default set to 80, can be any multiple of 20Buffer Mask
: any value between 0-15, according to coverage of cloud mask
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:52:01Z ----------------------------------------------------------------
The saved raster function can then be exported as an rft(raster function template). rft is used in the Generate Raster From Raster Function
tool to get the final results, as this tool is designed for raster processing using multiple threads to help speed up the processing.
View / edit / reply to this conversation on ReviewNB
BP-Ent commented on 2022-12-28T18:52:02Z ----------------------------------------------------------------
In this notebook, we have demonstrated a method to remove clouds from satellite imagery with a combination of a pre-trained deep learning model and Esri raster functions. Further, it demonstrated how you can use your developed python raster functions in ArcGIS Pro.
Thanks @BP-Ent, updated as per review
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-30T04:38:07Z ----------------------------------------------------------------
you have uploaded data using geosaurus org which is in ArcGIS Online org. However, the naming of the variable name is ent_gis which means it is enterprise portal. The naming of the variable is incorrect.
remote ent_gis line and use gis variable in the cells that get an item
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-30T04:38:08Z ----------------------------------------------------------------
instead of ent_gis, use gis
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-30T04:38:09Z ----------------------------------------------------------------
instead of ent_gis, use gis
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2022-12-30T04:38:09Z ----------------------------------------------------------------
item url is deldevs, which is private org
@sdash77 @jyaistMap FYI this notebook needs re-work, Deferred it to next release
@sdash77 this workflow has missing steps about how the cloud mask was generated from deep learning
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2023-11-22T04:14:04Z ----------------------------------------------------------------
The goal of this notebook is to detect and replace the cloud-contaminated region of the satellite images. Clouds often create obstructions in satellite imagery that can hinder results in processes like land cover classification and time series analysis. From a general perspective, this can be seen as an image denoising problem, where clouds are considered as noise in relation to the land surface in images. In this notebook, we will detect and replace cloud-contaminated regions of satellite imagery by combining a pretrained deep learning model, available in Living Atlas, with an image processing technique available in raster functions.
Hyperlink Living Atlas
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2023-11-22T04:14:05Z ----------------------------------------------------------------
Give a brief intro about the heading. create a flow chart that shows the workflow. Users may not understand why are we using a specific raster function after using a pretrained model.
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2023-11-22T04:14:06Z ----------------------------------------------------------------
remove this sub-heading
View / edit / reply to this conversation on ReviewNB
priyankatuteja commented on 2023-11-22T04:14:06Z ----------------------------------------------------------------
Next, we will use the Python raster function to blend areas using a Stepwise Local Radiometric Adjustment algorithm (Li et al., 2019). Stepwise Local Radiometric Adjustment is used to fill contaminated areas and is performed on each mask region of the target image. Using the replacement raster, the tool will replace pixels that are covered with clouds.
The raster functions can be downloaded as a python (.py) file named 'raster-functions' in the functions folder. The 'StepwiseLocalRadiometricAdjustment.py' file can be opened using 'Open Python Raster function'. Once opened, save it using 'Save As'. You can experiment with different values for the 'Size of Window' and 'Buffer Mask' parameters to achieve varying results based on your requirements and cloud coverage.