Support foreign keys in datasets

[Floppy]

In order to link data with reference data. Support for the latest version of JSON table schema will allow the schemas to reference other datasets, but we will need to make sure links are modified to work to other files in the same (or other) datasets.

[Floppy]

First step is to update to latest https://github.com/frictionlessdata/jsontableschema-rb, then we need to look at how foreign keys work with validation.

[jamesjefferies]

@pezholio I'm just getting my head around foreign keys in JSON schemas and I'm taking a look at this: https://specs.frictionlessdata.io/table-schema/ - what I can't quite work out is how you correctly reference another schema via a URI/L - I can see you have a reference section, with resource and fields.

Whereas in the jsontablesschema-rb gem - https://github.com/frictionlessdata/jsontableschema-rb there is a datapackage element too - are there any example schemas which reference another schema you can point me at? Thanks!

[jamesjefferies]

@pezholio ah - can I consider these examples valid? https://github.com/frictionlessdata/jsontableschema-rb/tree/master/spec/fixtures

and then looking at: https://github.com/frictionlessdata/jsontableschema-rb/blob/master/spec/fixtures/schema_valid_fk_string.json

Is it datapackage -> URI of datapackage - directory where datapackage.json exists?

then resource -> the name of the resource in the resources array in datapackage.json ?

and fields -> the name(s) of the fields specified in the fields array for the schema for the resource in datapackage.json ?

[pwalsh]

Hi @jamesjefferies

Note that the spec just moved to v1, and the ruby library has not been updated to support the most recent changes yet. We'll be talking with ODI (maintainers of the Ruby lib) in the coming weeks on the changes required to update to v1.

[pwalsh]

Also @jamesjefferies if you can supply us with an example of a dataset you have foreign keys in, I can also make sure to cross-reference this when we update the reference implementations (in Python and JavaScript).

[pezholio]

@jamesjefferies - Yeah that sounds right. Was going to point you to those specs. Just to confirm @pwalsh - are there likely to be any breaking changes moving to v1?

[pwalsh]

hey @pezholio yes, there are breaking changes. I hope to have the blog post and change log ready this week. I'll share with you as soon as I have a draft.

[pezholio]

OK, cool. It might be best to hold off on this feature (if possible) until the V1 changes are in the gem 👍

[jamesjefferies]

Ok thanks @pezholio @pwalsh - that's all really helpful - we'll pause this change until next week when we can see the breaking changes!

[jamesjefferies]

Hi @pwalsh is your blog post and change log ready yet please?

[quadrophobiac]

Aside from support for v1.0 of table-schema (and its attendant foreign key support) does CSVw (and thus CSVlint gem) support Foreign Keys? @Floppy @ldodds - in case either of you know off the top of your head?

[Stephen-Gates]

I think so... https://github.com/theodi/csvlint.rb/search?utf8=✓&q=foreign&type=

[ldodds]

Can't speak to csvlint specifically, but csvw does this, e.g:

https://www.w3.org/TR/tabular-data-primer/#enumeration-reference