arcgis-python-api icon indicating copy to clipboard operation
arcgis-python-api copied to clipboard

Flood mapping and monitoring using SAR data and deep learning sample notebook

Open shivanip32 opened this issue 2 years ago • 1 comments

Flood mapping and monitoring using SAR data and deep learning sample notebook


Checklist

Please go through each entry in the below checklist and mark an 'X' if that condition has been met. Every entry should be marked with an 'X' to be get the Pull Request approved.

  • [ ] All imports are in the first cell?
    • [ ] First block of imports are standard libraries
    • [ ] Second block are 3rd party libraries
    • [ ] Third block are all arcgis imports? Note that in some cases, for samples, it is a good idea to keep the imports next to where they are used, particularly for uncommonly used features that we want to highlight.
  • [ ] All GIS object instantiations are one of the following?
    • gis = GIS()
    • gis = GIS('home') or gis = GIS('pro')
    • gis = GIS(profile="your_online_portal")
    • gis = GIS(profile="your_enterprise_portal")
  • [ ] If this notebook requires setup or teardown, did you add the appropriate code to ./misc/setup.py and/or ./misc/teardown.py?
  • [ ] If this notebook references any portal items that need to be staged on AGOL/Python API playground, did you coordinate with a Python API team member to stage the item the correct way with the api_data_owner user?
  • [ ] If the notebook requires working with local data (such as CSV, FGDB, SHP, Raster files), upload the files as items to the Geosaurus Online Org using api_data_owner account and change the notebook to first download and unpack the files.
  • [ ] Code simplified & split out across multiple cells, useful comments?
  • [ ] Consistent voice/tense/narrative style? Thoroughly checked for typos?
  • [ ] All images used like <img src="base64str_here"> instead of <img src="https://some.url">? All map widgets contain a static image preview? (Call mapview_inst.take_screenshot() to do so)
  • [ ] All file paths are constructed in an OS-agnostic fashion with os.path.join()? (Instead of r"\foo\bar", os.path.join(os.path.sep, "foo", "bar"), etc.)
  • [ ] Is your code formatted using Jupyter Black? You can use Jupyter Black to format your code in the notebook.
  • [ ] IF YOU WANT THIS SAMPLE TO BE DISPLAYED ON THE DEVELOPERS.ARCGIS.COM WEBSITE, ping @ mohi9282 so he can add it to the list for the next deploy

shivanip32 avatar Sep 07 '22 05:09 shivanip32

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

  • the notebook looks good visually - I can't speak to technical content
  • the Introduction contains some extra characters...looks like the apostrophe in I'm came out jumbled. Can you look into the markdown cell and see if you can fix that?
  • intro_characters

jyaistMap avatar Nov 07 '22 22:11 jyaistMap

View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:06Z ----------------------------------------------------------------

Flooding is one of the most frequent and costly forms of natural disasters. They often strike without warning and can occur when large volumes of water fall in a short time, causing flash floods. Flood mapping is typically performed using the following methods:

  • Aerial observations
  • Ground surveys

However, when flooding is widespread, these methods become prohibitively expensive and time consuming. Furthermore, aerial observation and optical imagery can often prove difficult, if not impossible, due to obstructive weather conditions. During flooding conditions, clouds can prevent the use of optical satellite imagery for visualization and analysis. In these instances, synthetic-aperture radar (SAR) allows us to penetrate through clouds and hazy atmospheric conditions to continuously observe and map flooding.

In 2019, severe flooding occurred in the Midwest of the United States. Also known as the Great Flood of 2019, 14 million people were affected across multiple states. In this analysis, we will perform flood mapping and infrastructural inundation mapping of the St. Peters region of Missouri, which was one of the affected areas during the flood.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:07Z ----------------------------------------------------------------

Here, we convert the Sentinel-1 GRD VH polarization band to a 3 band raster using Export Raster. Under the Render Settings section, once Use Renderer is checked, Force RGB will be enabled.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:08Z ----------------------------------------------------------------

The resulting raster is generated from the Sentinel-1 GRD VH imagery using traditional histogram thresholding technique. The raster contains two classes, permanent waterbodies and flood water. This raster will be used as a Classified Raster in the Export Training Data Using Deep Learning tool.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:08Z ----------------------------------------------------------------

The feature layer contains two classes: 1 = Permanent Waterbodies and 2 = Flood Water. The feature layer will be used as the Input Feature Class in the Export Training Data For Deep Learning tool.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:09Z ----------------------------------------------------------------

The polygon feature class will be used as Input Mask Polygons in the Export Training Data For Deep Learning tool to delineate the area where image chips will be created.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:10Z ----------------------------------------------------------------

Next, we will utilize Jupyter Notebooks. Documentation on how to install and setup the necessary environment is available here.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:11Z ----------------------------------------------------------------

The prepare_data function takes a training data path as input and creates a fast.ai databunch with specified parameters, like transformation, batch size, split percentage, etc.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:12Z ----------------------------------------------------------------

As we have 2 classes (1=permanent waterbodies and 2=flood water) for this segmentation task, we need to perform an accuracy assessment for each class. To achieve this, ArcGIS API for Python provides the per_class_metrics function that calculates a precision, recall, and f1 score for each class.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:13Z ----------------------------------------------------------------

We will save the model that we trained as a 'Deep Learning Package' ('.dlpk' format). A Deep Learning Package is the standard format used to deploy deep learning models on the ArcGIS platform.

We will use the save() method to save the trained model. By default, it will be saved to the 'models' sub-folder within the training data folder.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:14Z ----------------------------------------------------------------

The resulting predictions are provided as a map for better visualization. The results show the spatial distribution of flood water in the Midwestern US during the 2019 floods. Sentinel-1 VV imagery of May 2019 are used for the analysis. In the map widgets, it can be seen that the trained UNetClassifier model is able to identify permanent waterbodies and flood water, as well as differentiate between the two. The deep blue color represents permanent waterbodies and the cyan color represents flood water.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:15Z ----------------------------------------------------------------

Three map widgets were created. The left widget displays natural color high resolution satellite imagery prior to flooding, the middle widget displays the sentinel-1 imagery during the flood event, and the right map widget displays the predictions of the trained UnetClassifier model. In the maps, St Louis city can be seen where the Illinois river and the Mississippi river converge. The model is able to identify river channels and differentiate from the flood water. The True Color Imagery can be used for visual interpretation for model accuracy.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:16Z ----------------------------------------------------------------

The pixel size of the raster is required to calculate the area of flood inundated areas. We will use the <raster>.properties.pixelSizeX and <raster>.properties.pixelSizeY functions to find the Pixel size of the raster.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:17Z ----------------------------------------------------------------

To calculate the area of land under flood water, we will use the <raster>.attribute_table() function to find the count of pixels per flood water class.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:17Z ----------------------------------------------------------------

This study requires the calculation of the area of land under flood water in terms of square km. The raster uses the projected coordinate system (3857), which has pixels in meters.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:18Z ----------------------------------------------------------------

The LULC raster for St. Louis is generated using the Land Cover Classification (Sentinel-2) pretrained model to assess the inundated areas per the LULC class.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:19Z ----------------------------------------------------------------

The flood raster will be converted to polygons using convert_raster_to_feature. We then use the import_data function to create a new feature layer containing the flood water polygons.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:20Z ----------------------------------------------------------------

Next, the LULC raster will be converted to polygons using convert_raster_to_feature. After getting the LULC polygons, we remove NODATA and Water polygons and use the import_data function to create a new feature layer containing the correct LULC polygons.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:21Z ----------------------------------------------------------------

To get the LULC classes for the flood inundated areas, we will use the overlay_layers function.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:22Z ----------------------------------------------------------------

After getting the LULC classes for the flood inundated areas, we will dissolve the polygons on the basis of gridcode. The output feature layer will have the combined area of each class in square miles units.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:23Z ----------------------------------------------------------------

The resulting feature layer has a column for the area per class, but the corresponding LULC class name is missing. We will add the class names to the dataframe using the code below.


View / edit / reply to this conversation on ReviewNB

BP-Ent commented on 2022-12-27T20:09:24Z ----------------------------------------------------------------

In this notebook, we have demonstrated how to use a UnetClassifier model with ArcGIS API for Python to extract flood water and permanent waterbodies. In Part 1, we covered how Sentinel-1 SAR data can be used for flood inundation mapping and monitoring. This process involved steps to prepare the input data, train a pixel-based classification model, visualize the results, generate accuracy metrics, and inferencing results on a test raster/area. Finally, in Part 2, we demonstrated the flood water inundated area in square kilometers and an infrastructural inundation assessment.