ncov-ingest
ncov-ingest copied to clipboard
ENH: Produce "small" metadata.tsv with only essential columns
Context
Our metadata.tsv is now so big that it becomes impossible to load into memory in full.
We should either create an "essential" metadata.tsv or in general partition metadata.tsv and make users merge later.
Right now we sort of require preprocessing with tsv-select
to pick relevant columns for interactive work (though augur filter seems to work fine with the metadata file as is so this isn't urgent).