Sentinel-1 SLC WCS problem / Atempt to serve Burst (parts) of Sentinel-1 via WCS

Open MathewNWSH opened this issue 1 year ago • 1 comments

Expected behavior and actual behavior.

Expected behavior

Burst Polygon, BBOX and Masking

Original data from Sentinel-1 SLC downloaded from here, specifically the file /...SAFE/measurement/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff single file download url should be able to be served via the WCS standard, in pieces defined by Bursts: For better understanding let's consider only one Burst, namely (the purple one): As you can see the shape is irregular, so as WCS bbox, would use this (the orange one): So I thought about using the BBOX symbolized by the orange polygon and then using the MASK created by the Burst vector layer, so the final output would consist only from data intersecting the orignal purple polygon :)

The POC Mapfile and VSIS3

Then the POC Mapfile would look like this:

MAP
    NAME "S1BURST"

    CONFIG "AWS_S3_ENDPOINT" "***"
    CONFIG "AWS_NO_SIGN_REQUEST" "YES"
    CONFIG "CPL_CURL_VERBOSE" "NO"
    CONFIG "AWS_HTTPS" "NO"
    CONFIG "AWS_VIRTUAL_HOSTING" "FALSE"
    CONFIG "GDAL_HTTP_TCP_KEEPALIVE" "YES"
    CONFIG "CPL_VSIL_CURL_CHUNK_SIZE" "98304"
    CONFIG "GDAL_INGESTED_BYTES_AT_OPEN" "16384"
    CONFIG "CPL_DEBUG" "OFF"
    CONFIG "GDAL_NUM_THREADS" "-1"
    CONFIG "PROJ_DEBUG" "OFF"

    EXTENT -98.9846740330880834 16.8257185042998358 -97.8401602114275590 18.4913746948899558

    MAXSIZE 250000
    SIZE 1024 1024
    IMAGETYPE PNG
    SHAPEPATH "/data/"
    PROJECTION
        "init=epsg:4326"
    END
    
    WEB
        IMAGEPATH "/tmp/"
        IMAGEURL "/tmp/"
        METADATA
          "wms_title"           "Burst Service"
          "wms_srs"             "EPSG:3857 EPSG:4326 EPSG:2180"
          "wms_enable_request"  "*"
          "wms_server_version" "1.3.0"
          "wms_feature_info_mime_type" "text/html"
          "wms_include_items" "all"
          "wms_getcapabilities_version" "1.3.0"
          "wcs_label"           "GMap WCS Demo Server" ### required
        "wcs_description"     "Some text description of the service"
         "wcs_srs"             "EPSG:4326"
         "wcs_timeitem" "timestamp"
         "wcs_metadatalink_format"  "text/plain"
         "wcs_enable_request"           "*"
        END
    END

    OUTPUTFORMAT
      NAME "GTiff"
      DRIVER GDAL/GTiff
      MIMETYPE "image/tiff"
      IMAGEMODE BYTE
      EXTENSION "tif"
    END



############################################################################
#  Tile Index
 
LAYER
 
    DEBUG 5
    STATUS OFF
    NAME "JanekBurst"
    TYPE POLYGON
    CONNECTIONTYPE OGR
    CONNECTION "/etc/mapserver/JanekBurst.gpkg"
    DATA "JanekBurst"
     
    PROJECTION
       "init=epsg:4326"
    END
     
    CLASS
        STYLE
         COLOR 0 0 0
        END
    END       

    METADATA
       "wms_enable_request" "!*"
    END
    PROCESSING "CLOSE_CONNECTION=DEFER"
    END
############################################################################

    LAYER
        DEBUG 5
        NAME "burst"
        TYPE RASTER
        STATUS ON
        DEBUG ON
        PROJECTION
           "+init=epsg:4326"
        END

        METADATA
           "wcs_label"           "Elevation/Bathymetry"  ### required
           "wcs_rangeset_name"   "Range 1"  ### required to support DescribeCoverage request
           "wcs_rangeset_label"  "My Label" ### required to support DescribeCoverage request
           "wcs_timeitem" "timestamp"
           "wms_enable_request" "!*"
           "wcs_bandcount" "1"
        END
       MASK "JanekBurst"

       #DATA "/etc/mapserver/test2.vrt"
       DATA "vrt:///vsis3/eodata/Sentinel-1/SAR/IW_SLC__1S/2024/05/10/S1A_IW_SLC__1SDV_20240510T004015_20240510T004042_053802_0689C7_59AA.SAFE/measurement/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff?a_srs=EPSG:4326"
       
        PROCESSING "CLOSE_CONNECTION=DEFER"
    END
END

As you can see to limit the level of complexity I'm not using tile index for now.

Actual behavior

VSIS3 attempt

The request:

http://localhost:8585/?map=/etc/mapserver/burst3.map&SERVICE=WCS&VERSION=1.0.0&REQUEST=GetCoverage&coverage=burst&crs=EPSG:4326&FORMAT=GTiff&BBOX=-98.7148208618164063,16.8216361999511719,-97.8398437500000000,17.1509628295898438&RESX=0.00006&RESY=0.00006

returns:

<?xml version='1.0' encoding="UTF-8" ?>
<ServiceExceptionReport version="1.2.0"
xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wcs/1.0.0/OGC-exception.xsd">
  <ServiceException code="NoApplicableCode" locator="bbox">msWCSGetCoverage(): WCS server error. Requested BBOX (-98.7147908618164,16.8216661999512,-97.83987375,17.1509328295898) is outside requested coverage BBOX (0,0,21673,13437)
  </ServiceException>
</ServiceExceptionReport>

So yeah, even if I use vrt://...?a_srs=EPSG:4326" GDAL/Mapserver is still using the pixel and lines based coordinate system (0,0,21673,13437)

Same situation if I add the &srcwin=0,0,21673,1493 to extract only the burst part of the image :(

<?xml version='1.0' encoding="UTF-8" ?>
<ServiceExceptionReport version="1.2.0"
xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wcs/1.0.0/OGC-exception.xsd">
  <ServiceException code="NoApplicableCode" locator="bbox">msWCSGetCoverage(): WCS server error. Requested BBOX (-98.7147908618164,16.8216661999512,-97.83987375,17.1509328295898) is outside requested coverage BBOX (0,0,21673,1493)
  </ServiceException>
</ServiceExceptionReport>

here the BBOX changed to the srcwin one, so the extraction worked, but still not being able to move from BBOX to pixels and lines ;/

VRT attempt

Unfortunately, the SLC data is georeferenced using GCP :/ I believe that the necessity of creating VRT files using gdalwarp -of VRT /vsis3/eodata/Sentinel-1/SAR/IW_SLC__1S/2024/05/10/S1A_IW_SLC__1SDV_20240510T004015_20240510T004042_053802_0689C7_59AA.SAFE/measurement/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff test2.vrt is extremely inconvenient, and in a production environment (aiming to serve bursts for the entire SLC history), it is an incredible obstacle. However, for testing purposes, I generated a VRT file using the above GDAL warp.

So I changed the DATA part to: DATA "/etc/mapserver/test2.vrt", and then made the request, but it returned a raster completely filled with 0s.

I decided to check what would happen if I disabled masking, so I changed MASK "JanekBurst" to #MASK "JanekBurst". Then I got a raster with values from 0 to 255, which looked as follows:

From what I understand, the MASK allows for rendering/issuing data that intersects only with the MASK layer. Therefore, either MapServer, despite the VRT, still doesn't recognize that both layers intersect, or the MASK in WCS is not functioning correctly (although its functionality has supposedly been implemented for a long time: https://github.com/MapServer/MapServer/pull/4469).

Questions

Is it possible to serve Sentinel-1 SLC data through WCS without the need to use gdalwarp -of VRT, for example, using a range defined by pixels and lines? I tried working with WCS 1.1.0 using this link: http://localhost:8585/?map=/etc/mapserver/burst3.map&SERVICE=WCS&VERSION=1.1.0&REQUEST=GetCoverage&IDENTIFIER=burst&FORMAT=image/tiff&BOUNDINGBOX=0,0,21673,1493,urn:ogc:def:crs:EPSG::4326

but it returns a weird file that I'm not quite sure what to do with: https://s3.waw3-2.cloudferro.com/swift/v1/slc/download%20(18)

Instead of using gdalwarp -of VRT, could I use a VRT connection string like vrtL//...? Perhaps there's a way to define the file parameters so that it can be retrieved normally.
Is it possible to serve data through WCS using pixels and lines? If so, what about the mandatory crs parameter?
Why values within downloaded file ranges from 0 to 255 while values of original file ale completely different? file download link: https://s3.waw3-2.cloudferro.com/swift/v1/slc/download%20(4).tiff
Why Mask for WCS is not working?

Burst metadata - Maybe it will help to better understand the idea of burst itself (metadata extracted from manifest.safe)

{'linesPerBurst': '1493'
'samplesPerBurst': '21673'
'platform': 's1a'
'subswath': 'iw1'
'polarization': 'vv'
'id': 1
'line': '0'
'azimuthTime': '2024-05-10T00:40:17.029758'
'sensingTime': '2024-05-10T00:40:18.164949'
'byteOffset': '107811'
'burstId_relative': '8689'
'burstId_absolute': '115560764'
'UR': [-98.68488000791601
16.82163658348603]
'UL': [-97.83984401471022
16.98157794240946]
'LL': [-97.85891995885116
17.1509623972803]
'LR': [-98.71480822683516
16.98927162918379]
'vrt': 'vrt:///vsis3/home/mniemyjski/S1A_IW_SLC__1SDV_20240510T004015_20240510T004042_053802_0689C7_59AA.SAFE/measurement/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff?a_nodata=0&srcwin=0,0,21673,1493'}

The value of key 'vrt' represents my first attempt on how to effectively use SLC without external VRT

All mandatory files to reproduce the problem

mentioned_VRT_and _GPKG_files.zip

SLC data downloaded from here: https://datahub.creodias.eu/odata/v1/Products?$filter=((Name%20eq%20%27S1A_IW_SLC__1SDV_20240510T004015_20240510T004042_053802_0689C7_59AA.SAFE%27))&$expand=Attributes&$expand=Assets

The single file on which the ticket is based on https://s3.waw3-2.cloudferro.com/swift/v1/slc/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff

Mapfile as above

weird file from WCS 1.1.0 https://s3.waw3-2.cloudferro.com/swift/v1/slc/download%20(18)

MapServer version and installation method

root@a970365a5e90:/etc/mapserver# mapserv -v
MapServer version 8.0.1 PROJ version 9.4 GDAL version 3.9 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=OGCAPI_SERVER SUPPORTS=FASTCGI SUPPORTS=GEOS SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE INPUT=FLATGEOBUF

Mpaserver run via modified mapserver camptocamp image (so it uses the newest gdal version) mapserver: image: camptocamp/mapserver:8.0-gdal3.8-arm64 ports: - 8585:80 volumes: - ./mapfiles:/etc/mapserver restart: unless-stopped environment: - MIN_PROCESSES=1 - MAX_PROCESSES=20 - IO_TIMEOUT=300 links: - database

May 28 '24 09:05 MathewNWSH

Also, the biggest question here is:

Is there any way of serving this kind of data (https://s3.waw3-2.cloudferro.com/swift/v1/slc/s1a-iw1-slc-vv-20240510t004017-20240510t004042-053802-0689c7-004.tiff) without needing to georeference them differently from the original data (GCPs)?

I realized that beyond the problem described above, I want to provide users with data in the same manner as the original file.

May 28 '24 13:05 MathewNWSH