geonames icon indicating copy to clipboard operation
geonames copied to clipboard

Remove streaming downloader

Open orangejulius opened this issue 5 years ago • 2 comments

In addition to a standard download, this importer supports streaming data directly from a ZIP archive over HTTP.

When we originally wrote it, we thought the idea of every part of Pelias supporting a "streaming" mode would lead to great functionality. In practice, it just made things brittle, and we generally prefer a discrete download step followed by an import process that does not use the network (except possibly to contact Elasticsearch).

Removing this importer's streaming mode would simplify the code and remove lots of issues we've had with streaming ZIP files, which are inherently hard to stream.

orangejulius avatar Feb 17 '20 22:02 orangejulius

Somehow I originally made this issue in the pelias/docker repository. It was supposed to go here, in the Geonames importer repository. :man_shrugging:

orangejulius avatar Feb 25 '20 16:02 orangejulius

Ooh transferring issues between repos... fancy!

missinglink avatar Feb 26 '20 12:02 missinglink