WIP Let GMT detect if DCW version on server has been updated

[PaulWessel]

For those who did not install DCW from a tarball or Linux distributions, GMT will download the latest from the gmt server. However, until this PR there was no mechanism to update files if we release a newer version, like we just did with 2.1.0.

This PR will check if the version in the cloud is fresher than the one in.gmt/geography/dcw , and if so we delete the local files so new ones can be downloaded. Note: This does not affect the installation of those that pre-install DCW from the tarball or via Linux distributions. They need to update via their distribution or get the latest tarball.

Here I have renamed my 2.1.0 so it could not be found and my ./gmt/geography/dcw contained old 2.0.1 files. I then run

gmt pscoast -RFR -JM15c -Baf -Gred -P -V > t.ps

which will need to access DCW. Here is what happens (I have removed unrelated messages:

pscoast [INFORMATION]:   -> Download cache file: VERSION
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/VERSION ...
pscoast [INFORMATION]: Refreshed DCW VERSION file /Users/pwessel/.gmt/geography/dcw/VERSION
pscoast [NOTICE]: Downloading dcw-countries.txt for the first time - be patient
pscoast [INFORMATION]:   -> Download cache file: dcw-countries
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/dcw-countries.txt ...
pscoast [INFORMATION]:   -> Download cache file: VERSION
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/VERSION ...
pscoast [INFORMATION]: Refreshed DCW VERSION file /Users/pwessel/.gmt/geography/dcw/VERSION
pscoast [NOTICE]: Downloading dcw-states.txt for the first time - be patient
pscoast [INFORMATION]:   -> Download cache file: dcw-states
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/dcw-states.txt ...
pscoast [NOTICE]: Downloading dcw-collections.txt for the first time - be patient
pscoast [INFORMATION]:   -> Download cache file: dcw-collections
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/dcw-collections.txt ...
pscoast [NOTICE]: Downloading dcw-gmt.nc for the first time - be patient
pscoast [INFORMATION]:   -> Download cache file: dcw-gmt
pscoast [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/geography/dcw/dcw-gmt.nc ...
pscoast [INFORMATION]: Using country and state data from dcw-gmt
pscoast [INFORMATION]: Title  : DCW-GMT - The Digital Chart of the World for the Generic Mapping Tools
pscoast [INFORMATION]: Source : Processed by the GMT Team, 2022-JAN-16
pscoast [INFORMATION]: Version: 2.1.0
pscoast [INFORMATION]: DCW version 2.1.0 requires GMT version 6.1.1 or later.
pscoast [INFORMATION]: France
pscoast [INFORMATION]: Region implied by DCW polygons is -5.14223/9.56155/41.3388/51.0928

Now my .gmt/geography/dcw contains the contents of the latest release, including dcw-collections.txt. Rerunning this command shows that the files are accepted and no new downloads take place:

pscoast [INFORMATION]: Using country and state data from dcw-gmt
pscoast [INFORMATION]: Title  : DCW-GMT - The Digital Chart of the World for the Generic Mapping Tools
pscoast [INFORMATION]: Source : Processed by the GMT Team, 2022-JAN-16
pscoast [INFORMATION]: Version: 2.1.0
pscoast [INFORMATION]: DCW version 2.1.0 requires GMT version 6.1.1 or later.
pscoast [INFORMATION]: France
pscoast [INFORMATION]: Region implied by DCW polygons is -5.14223/9.56155/41.3388/51.0928

Like for other remote files, we check the presence of and age of .gmt/geography/dcw/VERSION and if older than a day we refresh it. It fetches the 6 byte file from the server.

Note: I have labelled this WIP in case there are some unintended consequences - hence I hope you can give this a spin - you may need to hide or remove any DCW files installed from tarball or installers.

[anbj]

What if you have custom entries in dcw-collections.txt?

[PaulWessel]

You should have those in ./gmt/dcw.conf. The downloaded DCW files are in .gmt/geography/dcw so no collisions.

[maxrjones]

Could this be used to support the gmt get -Ddcw feature request (https://github.com/GenericMappingTools/gmt/issues/4040)?

[PaulWessel]

Yes, probably.

[PaulWessel]

We have to be careful not to over-complicate this. Consider those who install DCW from their Linux repo or our installers: We currently check that directory first for DCW and if found we do not go on and check if there might be a more recent version in the cloud. If we did, we certainly do not want to override the first directory which may be under system control w.r.t. permissions. So then what? Do we always check the cloud no matter what so we can inform the user (once a day) that they are behind, which also depends on the GMT version needed to handle the latest DCW). Or do we restrict the update to those who installed from cloud in the first place only. Also if we found the cloud data is more recent than what the repo/installer placed, and if we therefore downloaded into .gmt/geograpjhy/dcw, do we now have to always check all these places to find the most recent DCW before we open files? I.e., open all those dcw-gmt.nc files, get version, and go with the most recent. Who amongst you want to code all this up?

I think there are legitimate reasons why many will want to install dcw-gmt from a repo or installer (permission issues, automation, local restrictions). These people may not want to have GMT auto-update anything. Those who want the latest could remove any dcw-gmt they have install from repos or installers and let GMT handle things.

I suspect none of this is clearly documented, and not sure if this behavior is ideal anyway. Happy to take comments on this.

[Esteban82]

I am bit lost. I have Ubuntu. I can't test this, right?

[PaulWessel]

You can, since you are build GMT on ubuntu, not installing gmt6.x.x form the ubuntu bistro. But you would have to do the same I did to test:

Build the branch normally - this will then use your DCW installation. Temporarily change the name of the directory with the DCW so that GMT cannot find them anymore Next time DCW is requested by pscoast or -R it will download from the server and store in ~/.gmt/geography/dcw.

[anbj]

I don't have any strong opinions about this, but (mixed bag);

• I like the idea that gmt takes care of 'it's own' dcw and gshhg. gmt downloads during install, puts them in .gmt and the user gets a notice when there is a new version (fear: bad latency cause timeout and annoying checks, everyday).
• don't rely too much on distributions own packages of gshhg and dcw - I think some of them can lag pretty far behind current versions
• always use that in .gmt if it exists. Checking multiple paths seems like a waste of cycles. Make the user aware of this in docs; gmt will use what is in .gmt - make sure that what is located there is up to date
• alternatively; Super KISS (and simple code!). Get tarballs yourself and show gmt the path. If you want to be up-to-date, check the tarballs.

[joa-quim]

I like the idea that gmt takes care of 'it's own' dcw and gshhg. gmt downloads during install,

It doesn't. It would be only during runtime.

do we now have to always check all these places to find the most recent DCW before we open files? I.e., open all those dcw-gmt.nc files, get version, and go with the most recent.

No.

[maxrjones]

do we restrict the update to those who installed from cloud in the first place only.

Yes, I think we can assume that anyone who choses a different path than having GMT auto-download to .gmt/geography does not want notifications/actions regarding dataset updates.

[PaulWessel]

OK, this PR is starting to get old (Jan 18). I suggest we start with simply letting cloud-derived DCW be updated and anyone with other forms for dcw installations shall deal with those manually (or via their Linux repo). That is the extent of this PR.

Enabling gmt get -Ddcw can be done if we actually act on #4040.

[Esteban82]

I try it on ubuntu and failed.

pscoast [INFORMATION]:   -> Download cache file: VERSION
pscoast [INFORMATION]: Downloading file http://test.generic-mapping-tools.org/geography/dcw/VERSION ...
pscoast [ERROR]: Libcurl Error: HTTP response code said error
pscoast [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
pscoast [INFORMATION]: Refreshed DCW VERSION file /home/federico/.gmt/geography/dcw/VERSION
pscoast [NOTICE]: Downloading dcw-countries.txt for the first time - be patient
pscoast [INFORMATION]:   -> Download cache file: dcw-countries
pscoast [INFORMATION]: Downloading file http://test.generic-mapping-tools.org/geography/dcw/dcw-countries.txt ...
pscoast [ERROR]: Libcurl Error: HTTP response code said error
pscoast [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
pscoast [ERROR]: Unable to open file /home/federico/.gmt/geography/dcw/dcw-countries.txt [permission trouble?]
pscoast [ERROR]: South is outside -90 to +90 degree range
pscoast [ERROR]: General map projection error

[anbj]

I just tried installing gmt without supplying paths to gshhg and dcw. It looks like gmt only fetches the resolution it needs for the current job, not the whole thing (i.e. all resolutions). This applies both when getting gshhg and dcw. I think #4040 is a good idea - and get all the files. better be done getting everything before your internet is gone..