food-inspector icon indicating copy to clipboard operation
food-inspector copied to clipboard

Make API importer only update new records

Open tamcap opened this issue 11 years ago • 0 comments

I am currently working on some improvements to the current API importer.

However, as it is right now, we import the whole database from Durham County - despite their API providing us with a field Update_Date and syntax to filter it by Update_Date__gt=mm/dd/yyy that we can use to significantly limit our data / cpu use. We should take advantage of it.

Implementation: we will need to run a query on our db before import starts to establish the "latest" Update_Date value for each table (establishments, inspections, violations) and then incorporate it into the query string.

tamcap avatar May 16 '14 04:05 tamcap