Custom dataset for audio scenario

[sonal-ssj]

I was looking if armory would have provision to evaluate system/defense on a custom dataset for audio ASR scenarios? Use case: We have a bunch of manually created adversarial examples (*.wav files) that we would like to evaluate our system on.

Edit: I understand that one way would be to write an attack class in armory that would do the attack, however I was looking for a quick way just to take it as a list of wav files.

Thanks for your consideration!

[davidslater]

Hi Sonal, most of our dataset functionality is a bit heavy-handed for what you're doing, as it's designed for easily transfer and replication on other systems, and would require creating a "dataset" file.

The adversarial attack custom class would be easiest, though the metrics might come out wonky.

After this release, we'll be revamping our datasets and making what you are asking for easier, and you have sparked an idea on my side on how to make this work much better, but we're not quite there for just taking a list of files.

[sonal-ssj]

Thanks, looking forward to the updates!