rnnoise
rnnoise copied to clipboard
xiph
Decoding the model
When I try to use the provided python script dump_rnn.py to decode the newweights9i.hdf5 model, I found that it can not work well. So I change a lot of it to make it work well. I am not sure if it is right in my way. Here I want share them to you. If you are not busy in some time, please help me check it. I have try it to decode the model i got. I add below in the begging.
from keras.constraints import Constraint def mean_squared_sqrt_error(y_true, y_pred): return K.mean(K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)
def my_crossentropy(y_true, y_pred): return K.mean(2*K.abs(y_true-0.5) * K.binary_crossentropy(y_pred, y_true), axis=-1)
def mymask(y_true): return K.minimum(y_true+1., 1.)
def msse(y_true, y_pred): return K.mean(mymask(y_true) * K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)
def mycost(y_true, y_pred): return K.mean(mymask(y_true) * (10K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true))) + K.square(K.sqrt(y_pred) - K.sqrt(y_true)) + 0.01K.binary_crossentropy(y_pred, y_true)), axis=-1)
def my_accuracy(y_true, y_pred): return K.mean(2*K.abs(y_true-0.5) * K.equal(y_true, K.round(y_pred)), axis=-1)
class WeightClip(Constraint): def init(self, c=2,name='WeightClip'): self.c = c
def __call__(self, p):
#return {'name': self.__class__.__name__, 'c': self.c}
return K.clip(p, -self.c, self.c)
def get_config(self):
return {'name': self.__class__.__name__, 'c': self.c}
add an argument to name = 'WeightClip' init
and change load_model from model = load_model('./newweights9i.h5', custom_objects={'msse': mean_squared_sqrt_error, 'mean_squared_sqrt_error':mean_squared_sqrt_error, 'my_crossentropy':mean_squared_sqrt_error, 'mycost':mean_squared_sqrt_error, 'WeightClip':foo})
to model = load_model(sys.argv[1], custom_objects={'msse':msse, 'mean_squared_sqrt_error': mean_squared_sqrt_error, 'my_crossentropy':my_crossentropy, 'mycost':mycost, 'WeightClip':WeightClip})
Hi! Thanks for the sharing. I am also learning about the project and when I try to get the denoise_data9.h5 using the bin2hdf5.py, something wrong happens. Would you please show me the way to use the bin2hdf5.py? Thanks.
@zhaoforever you can use the main function in denoise.c to get the feature for tranning.
@liuanping so you've been able to train a model with my code -- cool! As for loading it, I've had all kinds of problems getting Keras to do that. I'll try the change you suggested (once I'm done with the ICASSP paper).
@jmvalin i have trained a model to try do ASR in noisy, and the result is not so good. I think is because of the bandwidth in the high frequency band is little big. So i also try to use a 40 band feature to train a new model and have a try and i constrain the frequency from 0Hz-8KHz, because the frequency sample is 16KHz. I hope it would be better.
@liuanping I think 16KHz sampling rate will be enough for both ASR task and communication.
@zhaoforever I found that the speech information have been hurt in the high frequency band, this may damage the ASR because the machine can hear the high frequency information. So i want to save the high frequency information by using much more band like 40 instead 22
@liuanping As @jmvalin mentioned, this project mainly targets speech communication instead of speech recognition. So the algorithm architecture is based on some speech codec algorithm. I don't think the RNNoise will work well for ASR task without changing the sampling rate and band feathure. I think you are on the right way and looking forward to your results.
Besides , would you please share you email so we can have a deeper discussion.
Please keep the discussion public (if possible) so we can all benefit, thanks.
@liuanping Thanks for your Keras code and it works. But as you said, you change the band feature to 40 bands instead of 22. As I know, the features from the C code contains some algorithm from the opus, which means you have to make a lot of changes. Have you finish the test yet?
@yeah i have done it ,but it can not make better for ASR, some people advise me to train together with DNN or BLSTM at the same time. Also, i think this may heart voice. No i think it could use to combine with beamfoming tech.
First of all, thanks for publishing the RNNoise code...
I prepared speech_only.wav, noise_only.wav, speech_noise.wav. and then run
~/rnnoise$ git clone https://github.com/smallmuou/wavutils ~/rnnoise/wav$ ../wavutils/bin/wavinfo speech_only.wav ++++++++++++++++++++++++++++++++++++++++++++++
-
WAVEFORM INFORMATION +
++++++++++++++++++++++++++++++++++++++++++++++ Audio Format: 1 (0x0001) Num Channels: 1 (0x0001) Sample Rate: 48000 (0x0000bb80) Bits Per Sample: 16 (0x0010) PCM Size: 3564480 (0x003663c0)
and then,
step 1) ~/rnnoise/wavutils/bin/wav2pcm speech_only.wav speech_only.pcm ~/rnnoise/wavutils/bin/wav2pcm noise_only.wav noise_only.pcm ~/rnnoise/wavutils/bin/wav2pcm speech_noise.wav speech_noise.pcm
step 2) ~/rnnoise/src/denoise_training speech_only.pcm noise_only.pcm output.f32 -> I got "matrix size: 500000 x 87" message.
step 3) ~/rnnoise/training/bin2hdf5.py output.f32 500000 87 denoise_data9.h5
step 4) ~/rnnoise/training/rnn_train.py -> denoise_data9.h5 -> newweights9i.hdf5
step 5) ~/rnnoise/training/dump_rnn.py newweights9i.hdf5 rnn_data.c rnn_data.h dump_rnn.py.txt
step 6) make clean & make
step 7) ~/rnnoise/examples/rnnoise_demo speech_noise.pcm denoised_speech_noise.pcm
step 8) ~/rnnoise/wavutils/bin/pcm2wav 1 48000 16 denoised_speech_noise.pcm denoised_speech_noise.wav -> Successfully convert denoised_speech_noise.pcm to denoised_speech_noise.wav.
Good Luck ~~
From Jin
@Gram2017 Thanks for your experience about using the RNNoise. As you mentioned, the size of the speech_only.wav is 3564480,which seems to be a big data file. Does the file come from some public database? Thanks.
step 2) ~/rnnoise/src/denoise_training speech_only.pcm noise_only.pcm output.f32 -> I got "matrix size: 500000 x 87" message. ------------where is ''/rnnoise/src/denoise_training ''
@18307612949 ,generated by running './compile.sh' in the terminal at /src
@liuanping Hey, guy. How is your way to optimize this method for ASR? what improvement is useful to match ASR.
@Gram2017/everyone
The recipe in this comment shows how to generate the data file for a single wave file. Say I have a bunch of speech_only and noise_only PCM files. How do I prep the data for the trainer?
@Gram2017 Thanks for the description in detail. When I try to execute step 2, it's continuously running with terminal output as shown in screenshot attached. Do you have any suggestion to solve it ?
Hi，guys，learned many things from your discussion，wandering when I use a 16k hz training samples，do I need to change the code or not need to change the code?
The 'count' variable is set to a very high value in denoise.c and that's the reason for high execution time. Reducing the count value will help you see the output.
The output is meant to be redirected to a file and saved. The count variable just determines how many training samples are generated
I am getting below error while running "python dump_rnn.py newweights9i.hdf5 rnn_data.c rnn_data.h"
File "training/dump_rnn.py", line 88, in
I have also tried the code posted by @liuanping but again i am getting below error
File "training/dump_rnn.py", line 95, in
i have solved that error by just replacing init with init in @liuanping thank you for the code
@KrishnanParameswaran i replaced the init with init in following [image]
Tried to followboth @liuanping and @venkat-kittu suggestions, but I still have the following result: Can you please help me?
dfranceschini@PV-NB-DF03:TRAIN_10mins$ ../github/training/dump_rnn.py newweights9i.hdf5 rnn_data.c rnn_data.h
Using TensorFlow backend.
Traceback (most recent call last):
File "../github/training/dump_rnn.py", line 86, in <module>
model = load_model(sys.argv[1], custom_objects={'msse':msse, 'mean_squared_sqrt_error': mean_squared_sqrt_error, 'my_crossentropy':my_crossentropy, 'mycost':mycost, 'WeightClip':WeightClip})
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/models.py", line 270, in load_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/models.py", line 347, in model_from_config
return layer_module.deserialize(config, custom_objects=custom_objects)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 144, in deserialize_keras_object
list(custom_objects.items())))
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2525, in from_config
process_layer(layer_data)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2511, in process_layer
custom_objects=custom_objects)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 146, in deserialize_keras_object
return cls.from_config(config['config'])
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 1271, in from_config
return cls(**config)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/layers/core.py", line 851, in __init__
self.kernel_constraint = constraints.get(kernel_constraint)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/constraints.py", line 177, in get
return deserialize(identifier)
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/constraints.py", line 170, in deserialize
printable_module_name='constraint')
File "/home/dfranceschini/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 153, in deserialize_keras_object
return cls(**config['config'])
TypeError: object() takes no parameters
Has anyone fixed the below error in step 2? If so how? I tried reducing the count but that did not work. Thank you. Update: It is producing an array of size zero.
i would like to summarize few fixes that might need to be done for some of the issues highlighted in this forum
- for correcting junk characters on console output perform the following
Step-1: Edit rnnoise/src/denoise.c line 653 for the following #if 1 fwrite(features, sizeof(float), NB_FEATURES, fout); // instead stdout fwrite(g, sizeof(float), NB_BANDS, fout); fwrite(Ln, sizeof(float), NB_BANDS, fout); fwrite(&vad, sizeof(float), 1, fout);
NOTE: u may optionally reduce the count to lesser value from 50000000. i used 50000 for quick validation purposes. i guess 500000 samples should be good enough. please correct me if not
Step-2: run './compile.sh'
- TypeError: 'int' object is not callable
Follow the first comment by @liuanping in this discussion thread However, some minor corrections as follows (related to syntax corrections and indentation) NOTE: these are thrown as run time exceptions. so you will easily find out if missed.
Correction 1: missing multipler for 10K.square and 0.01K.binar_crossentrop in mycost method def mycost(y_true, y_pred): return K.mean(mymask(y_true) * (10 * K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true))) + K.square(K.sqrt(y_pred) - K.sqrt(y_true)) + 0.01 * K.binary_crossentropy(y_pred, y_true)), axis=-1)
Corretion 2: Invalid class initializer 'init'
class WeightClip(Constraint): def init(self, c=2,name='WeightClip'): self.c = c
Correction 3: Indentation issues if any
Should i have to change code if I train data with different sampling rate like 44100hz or 16khz.