Add datasets from data hub awesome collection

[henrykironde]

Collections

• Bibliographic data note: The raw dataset is XML, a good example of adding ingestion for xml formats see https://github.com/weecology/retriever/wiki/GSoC-2020-Project-Ideas#approach

  List of datapackages created

  • [ ] add

• Climate Change

  List of datapackages created

  • [ ] add

• Demographics (population)

  List of datapackages created

  • [ ] add

• Economic Data and Indicators

  List of datapackages created

  • [ ] add

• Education

  List of datapackages created

  • [ ] add

• Football

  List of datapackages created

  • [ ] add

• GeoJSON

  List of datapackages created

  • [ ] add

• Health Care Data

  List of datapackages created

  • [ ] add

• Inflation

  List of datapackages created

  • [ ] add

• Linked Open Data

  List of datapackages created

  • [ ] add

• Logistics

  List of datapackages created

  • [ ] add

• Machine Learning / Statistical

  List of datapackages created

  • [ ] add

• Movies and TV

  List of datapackages created

  • [ ] add

• Open Corporates

  List of datapackages created

  • [ ] add

• Property Prices

  List of datapackages created

  • [ ] add

• Reference Data

  List of datapackages created

  • [ ] add

• Stock Market Data

  List of datapackages created

  • [ ] add

• War and Peace

  List of datapackages created

  • [ ] add

• Wealth, Income and Inequality

  List of datapackages created

  • [ ] add

• World Bank

  List of datapackages created

  • [ ] add

• YAGO List of datapackages created

  • [ ] add

[mayurdeep]

Almost all the datasets at Datahub collection already uses its own datapackage.json to define the schema of the data. https://datahub.io/docs/core-data https://frictionlessdata.io/data-packages/

So can we not build a module that can parses datapackage.json and restructure it to our format.

I can work on that if this seems a good idea..

[henrykironde]

@pathak-mayurdeep, thats great. We should add all the datapackages here https://github.com/weecology/retriever/blob/master/scripts/datapackages.yml and then create the module

[ethanwhite]

@pathak-mayurdeep Here's our existing (unmerged) work from a couple of years ago on using existing Frictionless Data data packages: https://github.com/weecology/retriever/pull/980 Feel free to build off this if helpful.

[ethanwhite]

Or maybe this got included in #1010? @henrykironde - #980 suggests that everything went into #1010, but I don't remember us getting all the way to loading external packages. Do you remember the status of that work as of #1010?

[henrykironde]

We had reached a point where we could ingest some of the data but then specifications changed. Currently we need to create a dictionary for the data types.

[mayurdeep]

Thanks for all the info.. I'll start working on this.

[mayurdeep]

Sorry about the late PR, I was offline for several days due to some personal health issues..

I have used datapackage-py in this. Please let me know whatever changes is needed.