north america example includes tiger files for only a single county

[knaaptime]

I'm trying to build a pelias docker instance based on the north american example, but encountering several issues. First, the config file specifies only a single county for some reason. I'd be happy to do a PR to include every state, but if you try to download all of them using pelias download tiger, the census API will hit a rate limit and throw

Error: read ECONNRESET
  	at _errnoException (util.js:992:11)
   at TCP.onread (net.js:618:25)   

also this issue also happens repeatedly. If you try and run pelias download oa again, you get

error: [download] Failed to download data message=Command failed: unzip -o -qq -d /data/openaddresses /data/openaddresses/us-al-baldwin1gvIfbRknrmEj.zip
error:  cannot delete old /data/openaddresses/README.txt
       No such file or directory
, stack=Error: Command failed: unzip -o -qq -d /data/openaddresses /data/openaddresses/us-al-baldwin1gvIfbRknrmEj.zip
error:  cannot delete old /data/openaddresses/README.txt
       No such file or directory

obviously, you can delete the offending file download again, but it's a lot of trial and error (tiger files, on the other hand, seem to overwrite old versions just fine). Any thoughts on whether some of those errors could be warnings instead of crashing the build?

[missinglink]

Good catch, we'd be happy to merge a PR to include all tiger data.

I believe that omitting that config blocks defaults to downloading all data, as per the planet project.

[missinglink]

Those download errors are very annoying, for the census ftp site we used to have a mirror, I'll have to check if that's still running and up-to-date, we could switch to that.

For openaddresses and this long-standing yet intermittent unzip bug I would like to find a solution, preferably one which doesn't require downloading to disk first, or at least one which doesn't shell-out a bunch if commands to the shell and necessitate a large tmp volume.

[divE4pip]

I believe that omitting that config blocks defaults to downloading all data, as per the planet project.

@missinglink what are the downsides of downloading all the tiger data just for the north-america build (apart from the oblivious longer download time)?

Also what would I lose if I were to omit the tiger data?

Thanks in advance!

[mark-at-nuna]

Hi, I just ran into this issue today. I would also like to know what the consequences are of omitting Tiger data. Is it worse to have incomplete Tiger data, or none at all?

[missinglink]

If you change the config to match the planet project it will download TIGER data for all the states: https://github.com/pelias/docker/blob/master/projects/planet/pelias.json#L65-L71

Is it worse to have incomplete Tiger data, or none at all?

TIGER is only used for building the interpolation index, omitting TIGER will use address points from OpenStreetMap and OpenAddresses where available. Adding TIGER too will improve the interpolation accuracy but I can't give you an exact percentage.

Can someone please update the Washington County, OR section to match the planet file above and open a PR?

[orangejulius]

There are two reasons why I omitted most of the TIGER data when creating up the North America project:

• The US Census download server is very slow, so downloading the data takes a while
• The interpolation service takes an even longer time to process the data, so the pelias prepare all/interpolation step will take hours if all the data is included

Interpolation data is used only for the /v1/search endpoint, and (we suspect) makes only a marginal difference (maybe 10% or less?), so it's omitted in favor of making the project easier to get started with.

I'm open to ideas for messaging things better or finding a better balance between ease of setup and completeness.