data-releases
data-releases copied to clipboard
unitedstates
Create a Data Release Calendar
Per the "[NICAR-L] Data Release Calendar" thread on the mailing list a script that produced "a simple calendar display of some type" or an actual ical feed would be a useful addition to this project.
I needed a quick coding distraction Wednesday at lunch and decided to quickly throw a CSV to ical script together. Little did I know that @brentajones would come along with issue #3 and bring a schema into it. So, a lunch and an evening later I have churned out a basic script that can create ical's for us.
I will make this a pull request once I am done adding inline comments, etc. I wanted to make sure I informed people that I had this moving along / functional so that no one else pushed ahead on it in vain.
Basic Functionality:
- The code can read from an individual CSV or merge a folder full of CSV's based upon the command options passed to it.
- This currently only parses a small number of columns (name, date, desription, agency_name, url, contact_name, contact_email) but can easily be expanded to support more, or different columns as they are needed.
- The validation & converting relies on a set of objects built our of the Project Open Data Metadata Schema v1.1. These objects are constructed by an outside function that can be easily updated to add additional properties.
- Most of the schema functionality has been only stubbed out so that we can add in more refined parsing and checking later. The objects are built to have the properties that they need (except Spatial which I will let someone else handle)) without doing anything more than the most cursory testing of any of the properties that are not currently being used. This should make it easy to add functionality later as all one has to do is modify the setter of a property to convert the string to whatever format is needed. I have only created objects for the Dataset and its underlying Distribution, contactPoint, Publisher and Spatial properties.
- I also built a simple vagrant configuration & build script to allow people to test the data or get the ical without mucking with their machine.
This looks awesome, thank you, @elationfoundation! I wonder whether it should be within this repo or as a separate utility?
This is something I've been considering collating for a WHILE now. @dwillis. If you've seen http://databulltet.in, they do a nice job of a similar task but their range runs the gamut of topics and sources. Some sort of unified calendar with just official U.S. data releases would be helpful.
And yes, @elationfoundation, you're on the right track! Happy to help, time pemitting.
Just FYI: It seems that the energy behind collecting the links has died, and I don't plan on doing any additional work on this code unless this becomes a place where releases start to be stored.
Also http://databullet.in/ for those who want to click the above link.