Can't import custom data in pelias

[PhanSon95]

Hey guys, I have problem when import custom csv file. I've deploy pelias in k8s and I want import my custom csv file. When import custom csv, I got error :

unable to locate sqlite folder

Steps to Reproduce Steps to reproduce the behavior:

1. download code :

git clone https://github.com/pelias/csv-importer

2. add config file pelias.json in same directory with package.json

{
  "logger": {
    "level": "info",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      {
        "protocol": "http",
        "host": "localhost",
        "port": 9200
      }
    ]
  },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "imports": {
    "csv": {
      "datapath": "./data",
      "files": ["example.csv"]
    }
  }
}

3. add file example.csv in data folder

id,source,layer,name,housenumber,street,postcode,lat,lon,addendum_json_pelias
1,custom,address,Tòa nhà Hoàng Ngọc,82,Dịch Vọng Hậu,100000,21.0301873359366,105.785161098654,

4. Run command:

export PELIAS_CONFIG=./pelias.json
npm run start 

5. See error

/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/src/localPipResolver.js:19
      throw err;
      ^
unable to locate sqlite folder
Thrown at:
    at /home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/src/localPipResolver.js:19:7
    at createPIPService (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/src/pip/index.js:47:12)
    at LocalPipService (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/src/localPipResolver.js:17:3)
    at module.exports (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/src/localPipResolver.js:99:10)
    at localResolver (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/index.js:28:43)
    at resolver (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/index.js:22:12)
    at create (/home/mypc/workingcsv-importer/node_modules/pelias-wof-admin-lookup/index.js:11:42)
    at createFullImportPipeline (/home/mypc/workingcsv-importer/lib/importPipeline.js:13:23)
    at /home/mypc/workingcsv-importer/import.js:37:18
    at Module._compile (node:internal/modules/cjs/loader:1155:14)

Expected behavior import success custom csv data

Environment (please complete the following information):

• OS: Ubuntu 20.04 LTS

[PhanSon95]

Pelias in k8s already have data. But I want import more custom data and it not working

[missinglink]

The whosonfirst (WOF) source is required as it provides all the admin geometries which form the political hierarchy. If you previously imported WOF then you'll need to make sure that the data downloaded at that time is still accessible to your CSV importer.

[PhanSon95]

hi @missinglink Because of previously imported data too large. So I want to import custom csv data independently in this case It possible to import independently custom data?

[missinglink]

yes

[missinglink]

As mentioned above whosonfirst is strictly required, you may either download it again (easy option) or use the same data you downloaded last time (advanced option).

Steps:

• add the imports.whosonfirst section back in your pelias.json file.
• run pelias download wof

Try your import again pelias import csv and it will succeed.

[PhanSon95]

I've update my pelias.json and when run npm run start , I don't got error But when search data , I can't find new data import by csv This is my new pelias.json

{
  "logger": {
    "level": "debug",
    "timestamp": false
  },
  "esclient": {
    "apiVersion": "7.5",
    "hosts": [
      {
        "protocol": "http",
        "host": "localhost",
        "port": 9200
      }
    ]
  },
    "api": {
      "targets": {
        "csv": ["address"]
      }
    },
  "elasticsearch": {
    "settings": {
      "index": {
        "refresh_interval": "10s",
        "number_of_replicas": "0",
        "number_of_shards": "1"
      }
    }
  },
  "imports": {
    "whosonfirst": {
      "datapath": "/home/ubuntu/data/whosonfirst",
      "importPostalcodes": true
    },
    "csv": {
      "datapath": "./data",
      "files": ["example.csv"]
    }
  }
}

File example.csv

id,source,layer,name,housenumber,street,postcode,lat,lon,addendum_json_pelias
1,custom,address,Tòa nhà Hoàng Ngọc,82,Dịch Vọng Hậu,100000,21.0301873359366,105.785161098654,

But when search with elastic, I can't find data inserted http://localhost:9200/pelias/_search?q=layer:%22address%22