crump icon indicating copy to clipboard operation
crump copied to clipboard

Modify parser to ingest new CSV format

Open waldoj opened this issue 9 years ago • 2 comments

As of August 1, the SCC is publishing CSV instead of their gnarly fixed-width format. As of October 31, they will no longer publish the old format. So that provides a 3-month window in which to upgrade Crump to use CSV. That ought to mostly involve simplifying it a great deal.

waldoj avatar Jul 25 '16 19:07 waldoj

Next up: modify the file iterator so that, instead of ingesting them naively, they are ingested as CSV. Then modify:

line[name] = current_line[1][start:end].strip()

to instead strip and rename the field, but otherwise leave it alone.

waldoj avatar Oct 31 '16 02:10 waldoj

Now the problem is that CSV and JSON are both being streamed into a single file.

waldoj avatar Nov 08 '16 04:11 waldoj