datapusher icon indicating copy to clipboard operation
datapusher copied to clipboard

datapusher appending rows to existing datastore

Open Mbrownshoes opened this issue 9 years ago • 0 comments

I'm updating a few resources nightly using the python ckanapi resource_update action. The resource is updating in the filestore but it we're seeing duplicate rows in the datastore, so if the resource has 1000 rows we often see 2000 in the data explorer and when using the datastore api. When we manually click the 'Upload to Datastore' button, we do not encounter this error.

In the datapusher jobs.py file there is reference to this error: ''' Delete existing datstore resource before proceeding. Otherwise 'datastore_create' will append to the existing datastore. And if the fields have significantly changed, it may also fail. '''

What is the difference in terms of the datastore usage between manually and automatically updating a resource? Is there a workaround for automatically updating the datastore?

Mbrownshoes avatar Apr 28 '16 22:04 Mbrownshoes