Prepare needed?

[ThreepE0]

Hello and THANK YOU for your amazing work here. Coming from Teddy's repo, and I notice that the instruction for prepare is missing from the instruction. Is something like the following needed before training?

python prepare.py --model models/ThreeE0/mymodel.ckpt data/sound_in.wav data/sound_out.wav

I also ran into an issue with my wav files; It might be worth noting in the readme that it's best if metadata is excluded. Working off wav files created in Reaper, the trainer chokes. I had to open them in Audition and uncheck "include metadata" though there may be a way to do this in Reaper alone.

Thanks again!

[GuitarML]

@ThreepE0 A recent update has included the data preparation in the train.py script, so it is no longer needed. It is still available to use as a separate command, like you noted above.

The metadata is a good thing to bring up, ideally we will add the ability to ignore the metadata automatically, but until then it will need to be excluded from the wav files manually. There is a way to render the wav files without metadata in Reaper, and I'll post a picture of how I do that here tonight. Basically do a File->Render, uncheck all the extra boxes, and make sure it's mono 32FP WAV format.

Glad you are enjoying the project! As an open source project it is a team effort, and I hope to keep improving it until it matches the quality and ease of use of something like a Quad Cortex or Kemper, but with the flexibility of running on any home recording setup on a budget.

[ThreepE0]

After my gushing, I finally get to the point below haha X-D

I can't describe adequately how excited I am for this project. Saying that it seems like the project I've been imagining for years has come out of the sky with halo'd angels singing would hardly be an exaggeration. And seeing your improvements and expansions on Teddy's work has come at the perfect time for me as well.

For a few years, I've had this concept in my mind as the perfect project for machine learning, but I've been too under-equipped and under-educated to get any productive work done.

This project has motivated and taught me a lot already, and I'm so excited for the future. So thank you again!

Thank you for your reply regarding prepare! It might still be worth mentioning on the readme I think, or maybe in a doc file that could accompany the repo files? I know it's better to keep things simple, but looking at this repo (selfishly) as a learning tool, seeing the discrete step of data prep is useful.

Also, I noticed that when I trained a wav file set yesterday, I was able to train even though the wav files were in the wrong format. Is there any checking done, or will an error throw, if a manual prep step isn't taken, and the files are in the wrong format? The resulting model's test wavs were empty, and the one of the test output files started with a startling click/pop. Now in the interest of full transparency, I have to say two things that are of note:

1. I'm not completely 100% sure that the files were in the wrong format. While I am fairly confident they were, as I created the wav files in reaper and had no way to remove the metadata, I have since deleted the wav files in question, and don't have them available to test a manual prepare.py execution with.

2. I mistakenly included reverb in my output wav file, so the training likely would have failed or been poor anyways. Being a newb, I'm not sure if this would result in an empty prediction wav output.

I can put these following items into new issues if you like:

Thinking about reverb, I'm wondering if training should be done on outputs that do not include any cab or mic emulation? I am training a file currently, and seem to be getting much better loss numbers with examples that have no cab emulation enabled.

Lastly, and importantly, the downloaded JUCE vst3 seems to completely lock up and have jittery performance in the current version of Reaper. I found this forum post about it, and they mentioned that OpenGL rendering may help: https://forum.juce.com/t/reaper-ui-freezes-with-plugin-open/43762

[mishushakov]

wow, thanks for the awesome writeup if you only could imagine how excited we are!

answering your questions:

prepare.py is gone because:

• Things should be more stupid
• It will become relevant in docker

i completely agree, the whole training process should be more documented good point, we should be checking the files before we train

timed effects can not be trained using this model

you should not judge the quality of models solely based on the numbers

off-topic: @GuitarML why don't we add the testimonials and media articles to guitarml.com (can you open-source the website?)

hope i could be helpful, thanks

[GuitarML]

@ThreepE0 Glad to hear you are so excited about the project! Here are the example render settings in Reaper (change the Bounds setting as applicable to your project):

Thats good info about Juce and OpenGL, I was not aware of that. Also I agree on mentioning prepare.py in the Readme as an option. As far as training from a mic, or with IR/cab modeling, the simpler you can make the sound wave, the easier it will be to train. Ideally you would capture the signal pre speaker, then you have the flexibility of having a core sound to add IR's to.

If you are interested in diving into the machine learning aspect of the code, we could definitely use help in refining the model and collecting data on different parameters and models. Also see the GuitarLSTM/SmartAmpPro repositories; they use a different model that has some advantages over WaveNet, and there is alot to learn and improve on there.

@mishushakov That is a good point about not judging models only on the loss number, and there might be a better way to calculate loss for this particular problem. Your ear should be the final judge of what sounds good! And the testimonials on the website is a good idea. It's very energizing to see that people are so interested in both the music and the technology sides of this. I'll consider open sourcing the site, web design is not my expertise and I could use help, especially if we want to host a model database.

[ThreepE0]

Thanks! I just re-exported the wavs and am trying as we speak.

Have you had any luck making the gpu work harder or faster? I have a 3090 that is being used, but it’s hovering between 15-30% usage

For reference, my pull of Teddy's repo uses 100% of the gpu while calculating

Not sure if I have something else going on, I've run the training about 4 times now and when I run predict, I get much quieter, lower gain sounding wav file sounds than I'd expect.

I'm running a training on Teddy's repo now using the original TS9 files to see what results I get there.

Using Cuda 11 and Python 3.8.7 in case that's relevant

On Tue, Feb 9, 2021 at 7:17 PM Keith Bloemer [email protected] wrote:

@ThreepE0 https://github.com/ThreepE0 Glad to hear you are so excited about the project! Here are the example render settings in Reaper (change the Bounds setting as applicable to your project):

[image: image] https://user-images.githubusercontent.com/32459398/107444429-2e8b8680-6b00-11eb-9a20-6674136664f4.png

Thats good info about Juce and OpenGL, I was not aware of that. Also I agree on mentioning prepare.py in the Readme as an option. As far as training from a mic, or with IR/cab modeling, the simpler you can make the sound wave, the easier it will be to train. Ideally you would capture the signal pre speaker, then you have the flexibility of having a core sound to add IR's to.

If you are interested in diving into the machine learning aspect of the code, we could definitely use help in refining the model and collecting data on different parameters and models. Also see the GuitarLSTM/SmartAmpPro repositories; they use a different model that has some advantages over WaveNet, and there is alot to learn and improve on there.

@mishushakov https://github.com/mishushakov That is a good point about not judging models only on the loss number, and there might be a better way to calculate loss for this particular problem. Your ear should be the final judge of what sounds good! And the testimonials on the website is a good idea. It's very energizing to see that people are so interested in both the music and the technology sides of this. I'll consider open sourcing the site, web design is not my expertise and I could use help, especially if we want to host a model database.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GuitarML/PedalNetRT/issues/21#issuecomment-776334617, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6OCQAPYYPOQUZ3YEJUT6DS6HGAZANCNFSM4XLLGR2Q .

[GuitarML]

That's interesting about the GPU usage, something that might contribute to that is the model parameters. I lowered the number of channels from 16 to 10, and I also use 10 conv1d layers with no repeats instead of (I think) 8 layers with 3 repeats (24 layers) as in the original pedalnet. Since it's performing less calculations per pass through the model, it wouldn't need as much processing power. Just a theory, we could test it by matching the params from Teddy's repo. It also might be why you're getting a quieter model in comparison. You might benefit from raising the max_epochs to 2000 or 2500, or raising the number of channels, if your computer can handle running the larger model in real time.

[ThreepE0]

I'm not sure how, but the GPU usage issue seems to have cleared itself up. It now spikes up to full usage when calculating, and I'm not sure what if anything I did to change it. Sorry for the confusion!

I got a pretty good result after last night's run. I can definitely get it better, I just have to start with a better preset I think I was rushing. Take a look at the my...pluscab wav file in the zip.

I included the train.py file, I did pretty much what you suggested, trying to move back towards Teddy's repo a bit... I also ran 5000 epochs though, so I'm thinking that may be where most of the improvement came from? I'm going to try 5000 again but with your default settings to see how it's different. Could you take a look at the train.py file and let me know if you have any tips?

At a glance, it seems like the predicted output is now a small bit quieter, and maybe on a related note, transients are being flattened out. I'm trying not to rely too much on visuals, but I used your included files to train, and comparing ml_out.wav and ml_grano1_predict.wav, it does look like the transients might be flattened out, and the result sounds a bit less lively.

Keeping in mind that I have absolutely no idea what I'm doing, I'm wondering if it would make sense to analyze fewer samples at a time in order to capture transients?

If that would work, would I adjust sample_time or batch_size? sample_time appears to be calculated off sample rate, and it looks like it's set to .1 of the sample rate, so 4410 samples and about 10 milliseconds? Am I thinking about that right?

Btw should I keep posting here or is there a better way to share results?

https://www.dropbox.com/s/nnf4la7svg4rr0j/PedalNetRT-outputs.zip?dl=0

Thanks again so much for sharing this project, and I apologize for all the questions

[ThreepE0]

Just a quick update: I ran the same files again using your unchanged train.py with 5000 epochs, and I got better results. Now we're cooking! I'd be happy to share the resulting models and predict wave files if it would help in any way.

Could you take a look at my question regarding transients above and let me know what your thoughts are?

I'll try the LSTM repo again as well as the SmartAmpPro plugin again now that I have my apparent GPU issues sorted out and am getting better results in prediction. I was guessing that you could export json and use the model from that in these other plugins, but haven't had a second to try it.

I smashed together this export.py for the lstm model and it seems to generate a json file, but I haven't tried to load it yet:

` import tensorflow as tf from tensorflow.keras.models import load_model import tensorflow.keras.backend as K from tensorflow.keras.optimizers import Adam

import matplotlib.pyplot as plt import os from scipy import signal from scipy.io import wavfile import numpy as np import matplotlib.pyplot as plt import math import h5py import argparse

def pre_emphasis_filter(x, coeff=0.95): return tf.concat([x, x - coeff * x], 1)

def error_to_signal(y_true, y_pred): y_true, y_pred = pre_emphasis_filter(y_true), pre_emphasis_filter(y_pred) return K.sum(tf.pow(y_true - y_pred, 2), axis=0) / K.sum(tf.pow(y_true, 2), axis=0) + 1e-10

def normalize(data): data_max = max(data) data_min = min(data) data_norm = max(data_max,abs(data_min)) return data / data_norm

def savejson(args): ''' saves model to json ''' # Read the input_size from the .h5 model file f = h5py.File(args.model, 'a') input_size = f["info"]["input_size"][0] f.close()

# Load model from .h5 model file
name = args.out_filename
model = load_model(args.model, custom_objects={'error_to_signal' : error_to_signal})

print("creating .json file....")
model_json = model.to_json()
with open(args.out_filename, "w") as json_file:
    json_file.write(model_json)

if name == "main": parser = argparse.ArgumentParser() parser.add_argument("out_filename") parser.add_argument("model") parser.add_argument("--train_data", default="data.pickle") parser.add_argument("--batch_size", type=int, default=4096) args = parser.parse_args() savejson(args) `

I have some questions about how the plugin/model can adjust gain after the fact without modelling an amp on differing settings, but I don't want to be a bother.

Thanks again!

[GuitarML]

I don't mind the questions at all! This is a learning experience for everyone involved, myself included, so the more questions asked the better. Just be patient with me, I might not respond right away but I get notified of any comments so it's on my list. Hopefully will have time to answer all your questions later today, but briefly about the gain question; all it's doing is adding gain to the signal prior to the model. So the EQ and Gain let you modulate the snapshot model to some extent, but it is still just a snapshot of a single setting.

Once you're up to speed I'll close out this issue; and as an additional resource you can use the Facebook community page (see link on the SmartGuitarPedal GitHub project page). I need to advertise it more, but it's basically my easy solution for forum/discussion for users.

Here's the group link: https://facebook.com/groups/674031436584335/?ref=pages_profile_groups_tab

• I'm not able to get to the dropbox items, it says it has been deleted?

• yes, sample time is set for 44.1kHz audio, if that ever changes in the .wav, this setting needs to change too

• I have not done much testing with the batch_size, but I'm not sure if that would fix the transients. What exactly does that mean as far as sound if the transients aren't captured?

• The SmartAmp and SmartAmpPro models are incompatible with eachother since SmartAmp is WaveNet and the Pro is LSTM. What are you doing in the code you posted? I'm not 100% sure but I think the to_json method doesn't export weights, just the overall model structure.

[ThreepE0]

Thanks Keith. Btw I figured out why my GPU wasn't being used. Turns out there were two problems:

1. windows task manager just doesn't show my GPU usage properly. I'm used to watching my GPU usage in hwinfo and rainmeter, so the day I just happened to be looking at task manager instead, I was confused when I saw 0% usage. Looking at it with fresh eyes, hwinfo shows 99% core usage when running train.py while task manager shows 0%.

2. I'm not sure if this turned out to be relevant for this project, but I'll mention it anyways: The tensorflow-gpu version that I had installed was not compatible with the version of CUDA and Cudann I had installed. This is the kind of thing I'd set up once and forget to take notes on, but I just had to go through it again on another PC, so I was able to nail it down.

On Fri, Feb 12, 2021 at 6:42 AM Keith Bloemer [email protected] wrote:

I don't mind the questions at all! This is a learning experience for everyone involved, myself included, so the more questions asked the better. Just be patient with me, I might not respond right away but I get notified of any comments so it's on my list. Hopefully will have time to answer all your questions later today, but briefly about the gain question; all it's doing is adding gain to the signal prior to the model. So the EQ and Gain let you modulate the snapshot model to some extent, but it is still just a snapshot of a single setting.

Once you're up to speed I'll close out this issue; and as an additional resource you can use the Facebook community page (see link on the SmartGuitarPedal GitHub project page). I need to advertise it more, but it's basically my easy solution for forum/discussion for users.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GuitarML/PedalNetRT/issues/21#issuecomment-778146283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6OCQDFHMT3HEEVTKLLXB3S6UH2ZANCNFSM4XLLGR2Q .