hepcrawl
hepcrawl copied to clipboard
inspirehep
post-enhancement: complete CC-license information
Create a workflow task to complete creativ commons license information which could be added to POSTENHANCE_RECORD for HEP records.
Expected Behavior
In the HEP schema the license contains the license and the url. For CC licenses this is redundant and often only one of the two is contained in the original metadata. Instead of completing it in all the individual crawlers this task could be done in a central place.
I think the literature builder https://github.com/inspirehep/inspire-schemas/blob/36bb1791b4df5890e5445f850c59ed9c5ee9b7c9/inspire_schemas/builders/literature.py#L493-L519 is a better place for this, but I agree in the principle of centralizing this normalization.
Obviously you know the system better than me. Does every record 'pass' literature.py? Also user suggestions and new records by BibEdit?
all new records go through the builder. Migrations from legacy and manual edits using the record editor don't. Do you think it's needed there too?
all new records go through the builder
@michamos besides records we will gather from DESY interim harvester and CDS.
Maybe that at some point we should ditch inspire-dojson and write a driver that transform bibrec from MARCXML using the Builder.
Maybe that at some point we should ditch inspire-dojson and write a driver that transform bibrec from MARCXML using the Builder.
Probably not, because then you will have to reimplement all the normalization that handles anomalies in Legacy's data (you have to ensure that you don't regress on the ~500 test cases in inspire-dojson).
The direction URL -> license name has been added to the builder in inspirehep/inspire-schemas#244 and inspirehep/inspire-schemas#245. The other direction has not been implemented yet.