Is OpenDataCache still caching data.cityofnewyork.us

[chrismetcalf]

One of our users (@rschmukler) is looking for full exports of the NYC ACRIS datasets, and I recommended he check out OpenDataCache for some quick dumps. Unfortunately, it looks like everything on ODC for data.cityofnewyork.us is 3+ months old:

http://www.opendatacache.com/data.cityofnewyork.us/

Are you guys still crawling that data portal? Any particular reason those jobs are paused?

[rschmukler]

@talos any word on this?

[talos]

I restarted the jobs, but I noticed about four months ago that they were tending to die on their own, while for a long time they were very reliable without touching. @chrismetcalf do you know of any changes on Socrata's side in the last three to five months (maybe six, I haven't paid super close attention) that could cause this? Error logs have not been informative. Any rate limits or other alterations?

[rschmukler]

@talos I recently opened a ticket with Socrata re: these large downloads ending prematurely. They just let me know that they deployed a fix to production and indeed it seems to be working now. Not sure if it was the same issue that you're describing, but it could be worth kicking them off once more and seeing if it has resolved

[talos]

Cool. As far as I can tell everything is still running, but I'll keep my eyes on it.

[talos]

Looks like stuff is going down after a few days.

[chrismetcalf]

@talos What do you mean?