Permit forwarded WMS requests to be cached

[keithamoss]

I'm going to throw this out here and see what people think of the idea.

If MapCache forwards a request to a WMS service for a single tile (256x256 / 272x272) of a layer it would be great to be able to specify that that layer be stored in a cache. I'm envisioning something like this:

   <cache name="sqlite" type="sqlite3">
      <dbname_template>/ms4w/tmp/ms_tmp/mapcache/{layer}/{grid}.db</dbname_template>
      <hitstats>false</hitstats>
   </cache>

   <service type="wms" enabled="true">
       <forwarding_rule name="first rule">
        <cache>sqlite</cache>
        <grid>g</grid>
        <append_pathinfo>true</append_pathinfo>
        <http>
          <url>http://server/mapserver?</url>
        </http>
       </forwarding_rule>
   </service>

I see the grid as either being specified manually, as here, or there being some smarts behind it by examining the SRS provided and working out the grid to use from there.

For expiry it would be great if mapcache could examine the Cache-Control or Expires headers and use that to determine when it needs to bypass the cache and request a new copy.

Thoughts and revisions more than welcome.

[tbonfort]

I imagine that you are trying to build a set of caches without knowing the layer names in advance right ? As for the expiry handling, please open another ticket.

[keithamoss]

Yup, that's correct.

I've opened https://github.com/mapserver/mapcache/issues/36 for the expiry handling side of things.

Let us know if you'd like anything more by way of explanation or detail of how we see this working.

[tbonfort]

mapcache already has a built in "mirroring mode" when configured with a single wms source. It will cache all LAYERS combination on getmap requests that corresponds to a configured <grid>, and will forward the rest to the original wms server.

<?xml version="1.0" encoding="UTF-8"?>

<mapcache mode="combined_mirror">
   <cache name="disk" type="disk">
      <base>/tmp</base>
      <symlink_blank/>
   </cache>

   <source name="vmap0" type="wms">
      <getmap>
         <params>
            <FORMAT>image/jpeg</FORMAT>
         </params>
      </getmap>

      <http>
         <url>http://www2.dmsolutions.ca/cgi-bin/mswms_gmap</url>
      </http>
   </source>

   <tileset name="mirror">
      <source>vmap0</source>
      <cache>disk</cache>
      <grid>WGS84</grid>
      <grid>g</grid>
      <format>JPEG</format>
      <metatile>5 5</metatile>
      <metabuffer>10</metabuffer>
      <expires>3600</expires>
   </tileset>


   <default_format>JPEG</default_format>

   <service type="wms" enabled="true">
      <full_wms>assemble</full_wms>
      <resample_mode>bilinear</resample_mode>
      <format>JPEG</format>
      <maxsize>4096</maxsize>
      <forwarding_rule name="catch all">
            <http>
               <url>http://www2.dmsolutions.ca/cgi-bin/mswms_gmap</url>
            </http>
      </forwarding_rule>
   </service>
   <service type="wmts" enabled="true"/>
   <service type="tms" enabled="true"/>
   <service type="kml" enabled="true"/>
   <service type="gmaps" enabled="true"/>
   <service type="ve" enabled="true"/>
   <service type="demo" enabled="true"/>

   <errors>report</errors>
   <lock_dir>/tmp</lock_dir>

</mapcache>

The important part is mode="combined_mirror" which can also take the value split_mirror:

• In "combined_mirror" mode, a request with LAYERS=layer1,layer2 will cache a single tileset (i.e. the wms will be queried once per tile, with LAYERS=layer1,layer2).
• In "split_mirror", a request with LAYERS=layer1,layer2 will result in two tilesets being cached (i.e. the wms will be queried twice, once with LAYERS=layer1, and once with LAYERS=layer2)

[keithamoss]

Thanks Thomas, I didn't know this feature existed. It's pretty much exactly what we're after - though the ability to mirror more than one WMS source would be great.

[Manfred73]

Hi,

I am using above mapcache.xml, but when I make a request, I get an unsupported format error in the apache log file:

Request:

http://10.31.42.202:8008/mapcache/?map=/var/www/mapserver/maps/mymapfile.map&service=wms&request=GetMap&version=1.1.1&layers=wind_area&srs=EPSG:4326&BBOX=-43.6,17.5,42.8,73.6&WIDTH=400&HEIGHT=300&FORMAT=image/png&TIME=2016-01-01T18:00:00.000Z

Error:

[Wed Feb 10 12:24:50.149724 2016] [:error] [pid 15:tid 140199099021056] [client 10.31.42.169:59779] wms request for tileset wind_area returned an unsupported format:\n<HTML>\n<HEAD><TITLE>MapServer Message</TITLE></HEAD>\n\n<BODY BGCOLOR="#FFFFFF">\nmsCGILoadMap(): Web application error. CGI variable "map" is not set.\n</BODY></HTML>

When I do a request for GetCapabilities, this seems to work OK and it doesn't complain about the CGI variable 'map' is not set: http://10.31.42.202:8008/mapcache/?map=/var/www/mapserver/maps/mymapfile.map&service=wms&request=GetCapabilities

Any idea why the getMap request fails with this error?

Regards, Manfred

[sgrossberndt]

The definition of MAP must be part of the source->getmap->params for the mapserver to query, not part of the mapcache request. This question is not related to the issue, please create an own instead.

[sgrossberndt]

The mapcache mirror modes are still not part of the documentation.