physionet-build icon indicating copy to clipboard operation
physionet-build copied to clipboard

Expand Schema.org metadata to support Croissant 🥐

Open tompollard opened this issue 5 months ago • 1 comments

In https://github.com/MIT-LCP/physionet-build/pull/640 we added support for Schema.org metadata. Croissant is an extension to this schema that captures additional information for finding and using the underlying data (with a focus on machine learning research).

The Croissant project is hosted at: https://github.com/mlcommons/croissant. There are example datasets at: https://github.com/mlcommons/croissant/tree/main/datasets (see "metadata.json" for the relevant dataset).

There are notes on working with Croissant in TensorFlow at: https://www.tensorflow.org/datasets/format_specific_dataset_builders#croissantbuilder

We should consider expanding our Schema.org metadata to conform with the Croissant spec.

Geoff Thomas (Kaggle) suggested that we may encounter issues around file size (e.g. apparently metadata files can get very large, to the point where Google Datasets refuses to index them).

tompollard avatar Jan 24 '24 19:01 tompollard