scaper
scaper copied to clipboard
justinsalamon
Sources don't always sum to mix, issue with clipping?
"Sometimes the sources don't come that close to summing to the mixture after scapering -- and this is when not using reverb.
Some of this might be related to clipping, as in the mixture clips when the sources don't or vice versa. I guess I can just turn down the ref_db and the upper SNR range, but is there a better way than trial and error?"
After reducing the volume, and padding/clipping at the end, I eliminated the mismatch that was coming from clipping. There still seems to be another mismatch, that happens every so often (a few times per 1000) where it seems like one or more of the sources gets dropped. Anyway It's easy to check for them -- just generate 1000 mixtures, read in all the sources and background, add them up, and see how far from the mixture they are. Then pick out the outliers and make a scatter plots of the mixture against various sums of the sources.
Doing this on one example, I found out that the mixture was equal to the first foreground source --- no background and no second foreground source.
It seems to be associated with the off-by-one errors. One hypothesis could be that sox partially fails to add up the sources if you get an off-by-1 error. In the above case both the mixture and the particular source that ended up in it had +1 samples, whereas the ones that were dropped had the correct number of samples.
@epicycles are you able to reproduce the issue for the same set of files, or is it a heisenbug? If it's reproducible, any chance you could post a MWE here?
Since this seems to be is an issue with sox, I'm not sure we'll be able to properly fix this without getting rid of sox altogether?
I haven't tried exactly reproducing it, since I'm randomizing when I generate, but I can try that. It's something that seems to happen in a small percentage of the soundscapes though - in my case it happens in about 0.2%-0.3% of cases. I am generating 1000's of soundscapes 10s long with from 1 to four sources. With PR#55 the source files are available to check whether they add to the mixture, and I would recommend doing these checks on the output files in PR#55 anyway as a unit test for that PR. My current work-around is for each soundscape to read in the mixture file and all the associated output sources, pad/trim them to the nominal length to get rid of off-by-one-sample issues, then add up the source files and write the sum to the mixture file. I agree this will likely be solved by moving away from sox entirely. A less extensive change, in the meantime, could be to just replace the sox combiner part by reading and summing the intermediate source files in memory, before writing out the mixture file.
I can replicate the off by one error in the test cases (it fails the strict test, which just does a straight sum). To sum the intermediate source files in memory, I need something like soundfile or librosa accessible from scaper, which would mean making that a requirement. Leaning towards librosa if we do this so that we can eventually do #76.
Related to #73.
Hello,has this problem been solved? I had the same issue. I use scaper to create some mix audio data, but the sum of the isolated audio is not equal to the mix audio.
I believe this issue should have been resolved when we switched audio processing to in-memory, right @pseeth ?
@Onlytwow can you share your script and let us know what versions of Scaper, python, and your OS you're using? thanks
Oh, yeah this should be closed, I think. We now have fix_clipping=True, and we sum everything up in-memory. Let us know if it's broken in the most recent version of Scaper!
@justinsalamon Scaper:1.6.4 python:3.6.10 os:Ubuntu18.04 scaper_test.zip
This zip file include the data I used and the script