System requirements to reproduce the ICASSP result

• CPU with 8+ threads
• NVIDIA GPU with 11+ GB V-memory
• SSD free space 500+ GB for full-scale experiment
• tar extraction temporarily requires additional free space 440 GB.

Recommended batch-size for GPU

• The larger the BSZ, the higher the performance.
• To allow the use of a larger BSZ than actual GPU memory, one trick is to remove allow_gpu_memory_growth() from the run.py.

Requirements

• NVIDIA driver >= 450.80.02
• Docker > 20.0

Create

You can create an image through Dockerfile and environment.yml.

git clone https://github.com/mimbres/neural-audio-fp.git
cd neural-audio-fp
docker build -t neural-audio-fp .

Further information

• Intel CPU users can remove libopenblas from Dockerfile.
• Faiss and Numpy are optimized for Intel MKL.
• Image size is about 12 GB or 6.43 GB (compressed).
• To optimize GPU-based search speed, install from the source.

Requirements

• NVIDIA driver >= 450.80.02, CUDA >= 11.0 and cuDNN 8 (Compatiability)
• NVIDIA driver >= 440.33, CUDA == 10.2 and cuDNN 7 (Compatiability)
• Anaconda3 or Miniconda3 with Python >= 3.6

Create

After checking the requirements,

git clone https://github.com/mimbres/neural-audio-fp.git
cd neural-audio-fp
conda env create -f environment.yml
conda activate fp

# Python 3.8: installing in the same virtual environment
conda create -n YOUR_ENV_NAME 
conda install -c anaconda -c pytorch tensorflow=2.4.1=gpu_py38h8a7d6ce_0 cudatoolkit faiss-gpu=1.6.5
conda install pyyaml click matplotlib
conda install -c conda-forge librosa
pip install kapre wavio

• Try installing tensorflow and faiss-gpu=1.6.5 (not 1.7.1) in separate environments.

#After creating a tensorflow environment for training...
conda create -n YOUR_ENV_NAME
conda install -c pytorch faiss-gpu=1.6.5
conda install pyyaml, click

Now you can run search & evaluation by

python eval/eval_faiss.py --help

This dataset includes all music sources, background noises, impulse-reponses (IR) samples that can be used for reproducing the ICASSP results.

Directory location

The default directory of the dataset is ../neural-audio-fp-dataset. You can change the directory location by modifying config/default.yaml.

.
├── neural-audio-fp-dataset
└── neural-audio-fp

Structure of dataset

neural-audio-fp-dataset/
├── aug
│   ├── bg         <=== Audioset, Pub/cafe etc. for background noise mix
│   ├── ir         <=== IR data for microphone and room reverb simulatio
│   └── speech     <=== subset of common-voice, NOT USED IN THE PAPER RESULT
├── extras
│   └── fma_info   <=== Meta data for music sources.
└── music
    ├── test-dummy-db-100k-full  <== 100K songs of full-lengths
    ├── test-query-db-500-30s    <== 500 songs (30s) and 2K synthesized queries
    ├── train-10k-30s            <== 10K songs (30s) for training
    └── val-query-db-500-30s     <== 500 songs (30s) for validation/mini-search

The data format is 16-bit 8000 Hz PCM Mono WAV. README.md and LICENSE is included in the dataset for more details.

Install checksumdir.

pip install checksumdir

Compare checksum.

checksumdir -a md5 neural-audio-fp-dataset
# aa90a8fbd3e6f938cac220d8aefdb134

checksumdir -a sha1 neural-audio-fp-dataset
# 5bbeec7f5873d8e5619d6b0de87c90e180363863d

python run.py train CHECKPOINT_NAME CHECKPOINT_INDEX

• If CHECKPOINT_INDEX is not specified, the training will resume from the latest checkpoint.
• In default configuration, all checkpoints are stored in logs/checkpoint/CHECKPOINT_NAME/ckpt-CHECKPOINT_INDEX.index.

python run.py train CHECKPOINT --max_epoch=100 -c default

Notes:

• Check batch-size that fits on your device first.
• The default config is set TR_BATCH_SZ=120 with OPTIMIZER=Adam.
• For TR_BATCH_SZ >= 240, OPTIMIZER=LAMB is recommended.
• For TR_BATCH_SZ >= 1280, LR=1e-4 can be too small.
• In NTxent loss function, the best temperature parameter TAU is in the range of [0.05, 0.1].
• Augmentation strategy is quite important. This topic deserves further discussion.

The config file is located in config/CONFIG_NAME.yaml. You can edit directory location, data selection, hyperparameters for model and optimizer, batch-size, strategies for time-domain and spectral-domain augmentation chain, etc. After training, it is important to keep the config file in order to restore the model.

python run.py COMMAND -c CONFIG

When using generate command, it is important to use the same config that was used in training.

python run.py generate CHECKPOINT_NAME # from the latest checkpoint
python run.py generate CHECKPOINT_NAME CHECKPOINT_INDEX -c CONFIG_NAME

# Location of the generated fingerprint
.
└──logs
   └── emb
       └── CHECKPOINT_NAME
           └── CHECKPOINT_INDEX
               ├── db.mm
               ├── db_shape.npy
               ├── dummy_db.mm
               ├── dummy_db_shape.npy
               ├── query.mm
               └── query_shape.npy

By default config, generate will generate embeddings (or fingerprints) from 'dummy_db', test_query and test_db. The generated embeddings will be located in logs/emb/CHECKPOINT_NAME/CHECKPOINT_INDEX/**.mm and **.npy.

• dummy_db is generated from the 100K full-length dataset.
• In the DATASEL section of config, you can select options for a pair of db and query generation. The default is unseen_icassp, which uses a pre-defined test set.
• It is possilbe to generate only the db and query pairs by --skip_dummy option. This is a frequently used option to avoid overwriting the most time-consuming dummy_db fingerprints in every experiment.
• It is also possilbe to generate embeddings (or fingreprints) from your custom source.

python run.py generate --source SOURCE_ROOT_DIR --output FP_OUTPUT_DIR --skip_dummy # for custom audio source
python run.py generate --help # more details...

The following command will construct a faiss.index from the generated embeddings or fingerprints located at logs/emb/CHECKPOINT_NAME/CHECKPOINT_INDEX/.

# faiss-gpu
python run.py evaluate CHECKPOINT_NAME CHECKPOINT_INDEX [OPTIONS]

# faiss-cpu
python run.py evaluate CHECKPOINT_NAME CHECKPOINT_INDEX --nogpu

In addition, you can choose one of the --index_type (default is IVFPQ) from the table below:

python run.py evaluate CHECKPOINT_NAME CHECKPOINT_INDEX --index_type IVFPQ

Currently, few options for Faiss settings are available in run.py client. Instead, you can directly run:

python eval/eval_faiss.py EMB_DIR --index_type IVFPQ --kprobe 20 --nogpu
python eval/eval_faiss.py --help

Note that eval_faiss.py does not require Tensorflow.

Tensorboard is enabled by default in the ['TRAIN'] section of the config file.

# Run Tensorboard
tensorboard --logdir=logs/fit --port=8900 --host=0.0.0.0