Add natural earth data

[henrykironde]

https://www.naturalearthdata.com/downloads/10m-cultural-vectors/

[Nageshbansal]

Hey @henrykironde , i want to work on this issue , can you guide me how to proceed for this issue? i already downloaded the dataset but i'm not getting how to add this dataset/files.

[henrykironde]

@Nageshbansal thank you for the interest in working on this issue. Since these are vector files, once you have download the dataset, run retriever autocreate -dt vector -d path-to-the-directory-containing-the- data-files. This will give you a template that you can work with. Let me know how that goes

[Nageshbansal]

@henrykironde i tried this , and i'm getting some error in some files so working on that Thanks for letting me know

[Nageshbansal]

@henrykironde installed GDAL successfully , but still getting this error "Skipped file: ne_10m_admin_0_antarctic_claims.shp name 'ogr' is not defined Remove the file from the folder and try again"

[henrykironde]

Try runing imports from python and report back what you get.

➜  rdataretriever git:(main) ✗ python
Python 3.9.6 (default, Jun 29 2021, 05:25:02) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> from osgeo import ogr
>>> from osgeo import gdal
>>> 

[Nageshbansal]

Import error i.e osgeo not found. I have installed GDAL by using OSGeo4W installer

[Nageshbansal]

@henrykironde successfully created template of data.

[henrykironde]

Good, now you will populate the fields based on the appropriate meta data. For example from reading about, it takes me to the GitHub page where the files are hosted.https://github.com/nvkelso/natural-earth-vector/blob/master/LICENSE.md

From the repo I see they have a license file that states "non-exclusive license", I would use that in the license field. And so on.

When working with raster data, you need to download Qgis. This tools can help you look at the various data and even get more info from the properties.

[henrykironde]

[henrykironde]

[Nageshbansal]

Got it, will try to do it asap, Thanks for this information :)

[henrykironde]

You can drag and drop the .shp file into Qgis

[henrykironde]

@henrykironde, I have added the dataset, tested it by using PostGIS in qgis, out of 81 tables, 75 are able to add to the database, rest of 6 is throwing error .for e.g NOTICE: table "ne_10m_admin_0_countries_bdg" does not exist, skipping and when i tried to debug it i found out that this file ne_10m_admin_0_countries_bdg exist but it's not able to be read

Create a PR on the Retriever-recipes repo, I will look at it.

[Nageshbansal]

okay I will create a PR, and there is one thing I want to know that by using "extract_all": "True", it extracts shapefiles in /.retriever/raw_data/cultural-10m/10m-cultural/ folder due to which retriever is not able to get shapefiles can you tell me is there any way to extract files in /.retriever/raw_data/cultural-10m/ itself.

Also, I think 6 tables are left out because their schema holds some fields which have null or zero values and according to SQL syntax, they can't be null . for e.g. size attribute of FCLASS_US have value 0 in fields of the schema of ne_10m_admin_0_countries_bdg.

                  {
                       "name": "FCLASS_US",
                        "precision": 0,
                        "size": 0,
                        "type": "String"
                    },