Replace .csv in dcat api
Is your feature request related to a problem? Please describe. Currently, the dcat api script (dcat.py) outputs to a .csv which is then ingested into merge_data.py. This should be converted to a .json output to keep consistent with the rest of the project.
Describe the solution you'd like
- [ ] Change the output of dcat.py to be a .json file
- [ ] Update merge_data.py to ingest the new .json file
- [ ] format code with Black before PR
- [ ] update docs (architecture.md and datapipeline.png)
Describe alternatives you've considered None
Additional context Original ticket triggering this change is #163
please assign the issue.. would love to work on it!
@thiyagarajan17 I did some work on this to do some bug fixes and had forgotten this issue was assigned to you. I've made a start on it if you've got any comments/feedback?
https://github.com/OpenDataScotland/the_od_bods/tree/jack-dcat-improvements