Daniel Galvez

Galv/ia download unsupervised

1

Just do `pip install internetarchive` and run: ``` python download_items.py ``` You can copy download_items.py on its own without the rest of the repo. It is a standalone file. You...

Galv/setuptools

1

Remove lingvo, allow for setuptools to be used for installation. Just do: python setup.py develop And that's it.

Example: ``` key="38th_Old_Fashioned_Underhill_Harvest_Market-38th_Old_Fashioned_Underhill_Harvest_Market_mp3_0.flac" value="'s a great experience it's really fun and we love to see and all the local crafts and meeting people" ``` We probably want to avoid this.

Things to do before neurips

1

- [ ] Create two separate datasets to distribute, one CC-BY, one CC-BY-SA. - [ ] Rerun yamnet on the entire dataset. This means we need to make it more...

Truncating keys to length 100 sometimes causes duplication

1

I found this by diffing utt2spk and spk2utt in kaldi: Can I write tar files with no maximum file name length? ``` -Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented\ _People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac Talented_People_w\ ith_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_87.flac -Talented_People_with_Jeanne_Murphy_10_16-Talented_People_with_Jeanne_Murphy_10_16_HD_mp3_88.flac...

Now that we support creating tar files of our data, we want to filter the files into two types: - CC-BY-SA files - CC-BY and public domain files We can...

Add license headers

1

We should have Apache 2.0 license headers on the top of all of our files, but we have been a bit lazy about this. We ideally should also have a...

Implement text normalization

1

We are lacking text normalization at the moment, which qualifies in my book as a serious flaw. I recommend Sparrowhawk as a first step: https://github.com/google/sparrowhawk Ideally we would wrap its...

Do the same thing as described in appendix A here on our transcripts: https://arxiv.org/pdf/2005.14165.pdf