bids-validator icon indicating copy to clipboard operation
bids-validator copied to clipboard

Error for TSV files that are not valid UTF-8

Open effigies opened this issue 3 years ago • 1 comments
trafficstars

According to Tabular files:

TSV files MUST be in UTF-8 encoding.

We currently don't validate this, which leads to situations when data is encoded in, e.g., ISO-8859 (https://github.com/OpenNeuroOrg/openneuro/issues/2515).

This one's a bit of a double-edged sword as it requires reading the entirety of every TSV file, which we've largely avoided to now.

effigies avatar Feb 09 '22 20:02 effigies