audioread
audioread copied to clipboard
beetbox
NoBackendError despite a backend (specifically FFmpeg) being installed
Hello!
This is my first time posting an issue, so please cut me some slack if I'm not following some protocol!
I was trying to use python's librosa package on Windows 10 and encountered the following issue.
After running x, _ = librosa.load('data/fma_small/000/000002.mp3', sr = None) I receive this stack trace:
---------------------------------------------------------------------------
NoBackendError Traceback (most recent call last)
<ipython-input-2-15a8daa0e7fd> in <module>()
1 start, end = 7, 17
2 filename = utils.get_audio_path(AUDIO_DIR, 2)
----> 3 x, sr = librosa.load(filename, sr = None, mono = True)
Q:\Program Files\Anaconda3\lib\site-packages\librosa\core\audio.py in load(path, sr, mono, offset, duration, dtype, res_type)
105
106 y = []
--> 107 with audioread.audio_open(os.path.realpath(path)) as input_file:
108 sr_native = input_file.samplerate
109 n_channels = input_file.channels
Q:\Program Files\Anaconda3\lib\site-packages\audioread\__init__.py in audio_open(path)
112
113 # All backends failed!
--> 114 raise NoBackendError()
NoBackendError:
Out of curiosity, I also tried to run audioread.ffdec.FFmpegAudioFile('data/fma_small/000/000002.mp3') (since I know that FFmpeg is installed) but I received:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in __init__(self, filename, block_size)
126 stderr=subprocess.PIPE,
--> 127 stdin=self.devnull,
128 )
Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in popen_multiple(commands, command_args, *args, **kwargs)
88 try:
---> 89 return subprocess.Popen(cmd, *args, **kwargs)
90 except OSError:
Q:\Program Files\Anaconda3\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds)
946 errread, errwrite,
--> 947 restore_signals, start_new_session)
948 except:
Q:\Program Files\Anaconda3\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
1223 cwd,
-> 1224 startupinfo)
1225 finally:
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
NotInstalledError Traceback (most recent call last)
<ipython-input-6-be3b460211e8> in <module>()
----> 1 audioread.ffdec.FFmpegAudioFile(filename)
Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in __init__(self, filename, block_size)
129
130 except OSError:
--> 131 raise NotInstalledError()
132
133 finally:
NotInstalledError:
Again, I have FFmpeg installed, and ffmpeg works as a command in cmd.
Is anyone able to spot the problem?
Thanks!
Huh! That’s frustrating. Thank you for the detailed tracebacks. I’m guessing this has something to do with Windows executable search vagaries. Can you please try running this in a Python console?
>>> import subprocess
>>> subprocess.Popen(['ffmpeg'])
My best guess is that your cmd shell has the PATH variable extended to have ffmpeg available, but however your librosa installation somehow does not have the same PATH.
If I understand your guess correctly, then I think you're onto something: I ran what you asked, and subprocess.Popen(['ffmpeg']) returned exactly what running ffmpeg in cmd returns.
Got it. In that case, does it work to import audioread directly in the Python REPL and use the FFmpeg backend? If so, it may be necessary to examine the Windows environment settings on the program that’s using it indirectly.
I get the exact same error (via librosa.load as well) but it seems to happen randomly when loading several files in succession.
It's hard to reproduce, but it happens frequently (every 50 files or so). This is all that I'm doing:
import librosa as lr
for audio_path in audio_paths:
audio_samples = lr.load(audio_path, None)[0]
Could NoBackendError be more specific about what's actually breaking? Could it actually be obscuring an OutOfMemory or something?
No, we don't currently have more detail available in that exception. Perhaps you could try the same diagnosis steps as @uipo78 above—namely, directly using the backend you expect to exist, so you can see exactly where it's failing?
Also, does the exception happen reliably for a specific file? If so, it could be that whatever backend you're using fails to read that specific file.
I'm having this same problem with any file with PyCharm. I debugged a bit and tried a few things:
The problem happens in line 89 of ffdec.py but inside the subprocess code. For my experiment in my system that line translates to:
subprocess.Popen(['ffmpeg', '-i', "C:\\Users\\amarafioti\\PycharmProjects\\inpainting-similarity-graphs\\audio_files\\pop\\audio\\01-Sargon-Mindless.mp3", "-f", "s16le", "-"], stdin=devnull, stderr=-1, stdout=-1)
where -> devnull = open(os.devnull)
I changed the code in ffdec.py to:
for i, command in enumerate(commands):
cmd = [command] + command_args
try:
return subprocess.Popen(cmd, *args, **kwargs)
except OSError as e:
print(e)
if i == len(commands) - 1:
# No more commands to try.
raise
and got this traceback:
[WinError 2] The system cannot find the file specified
but stopping on the exception and doing os.path.isfile(path) returns True. Maybe that WinError refers to ffmpeg?
I then tried installing avconv as it is the other command that my audioread tries. It also failed to load it. I checked that both ffmpeg and avconv where installed both as user variables and system variables just in case but that didn't help.
I ended up just running my program in the console where it works perfectly. Did you also had this problem with PyCharm and/or an IDE?
That's strange! My best guess is that your PyCharm setup is controlling the PATH environment variable differently from the rest of your system (or the command line prompt, at least). That would mean that the ffmpeg or avconv command is not available for audioread to find it. Maybe that can be set explicitly in the IDE's project settings?
YES, "fixed" it.
Following this intuition of the PATH environment variable behaving differently from the rest of my system I just copied ffmpeg to the parent folder of the directory and it worked! And then I realized I hardly ever close PyCharm, so I restarted it and it works! This issue also happens with terminals (environment variables don't take effect unless you open a new one).
@uipo78 maybe you had this same problem?
@carlthome it could be an OutOfMemory. Audioread just expects an OSError and there are a whole bunch of different exceptions that are OSErrors. You could debug that line or just change the code with what I proposed and see what gets printed.
@carlthome it could be an OutOfMemory.
Yup, I think that was it actually. It would be nice if audioread gave clearer errors than NoBackendError.
Can you still reproduce the error and debug to see if that was it? You should only set a breakpoint on the exception and remove avconv from the commands in line 32 of ffdec.py
Yeah, it would be interesting to explore a redesign of the error messages. The tricky thing is that we want to support automatic fallback between different backends, so if one backend can't be used, we don't want to immediately stop with a backend-specific error. But perhaps "aggregate" errors like NoBackendError could accumulate some context from each backend that failed?
I'm experiencing difficulty related to this and created this SO post:
https://stackoverflow.com/questions/46774309/zipping-conda-environment-breaks-audioreads-backend-python-pyspark
Any guidance would be appreciated. Thanks.
I have solved my problem. The summary is to (a) close python and/or Idle, (b) run the appropriate installation packages, and (c) setting the environment variables for ffmpeg and appending the environment variable to the path at https://github.com/librosa/librosa/issues/743
Regards Anthony of Sydney NSW
@carlthome I am facing the exactly same error and its random as you mentioned. Do you know how to solve this issue?
Not really, no. I eventually started calling ffmpeg via subprocess.Popen and am pretty happy with more explicit control for my purposes.
import shlex
import subprocess
import numpy as np
def decoder(path, duration=3.0, channels=2, sample_rate=44100, dtype=np.float32):
formats = {np.int16: 's16le', np.float32: 'f32le'}
args = f'''
ffmpeg
-v error
-n
-i "{path}"
-f {formats[dtype]}
-ac {channels}
-ar {sample_rate}
-
'''
args = shlex.split(args)
buffer_size = int(channels * np.dtype(dtype).itemsize * duration * sample_rate)
with subprocess.Popen(args, stdout=subprocess.PIPE) as pipe:
while True:
buffer = pipe.stdout.read(buffer_size)
audio = np.frombuffer(buffer, dtype=dtype)
audio = np.reshape(audio, (-1, channels))
yield audio
if len(buffer) < buffer_size:
break
for x in decoder('piano.wav'):
print(x.shape)
from IPython.display import Audio
audio = np.concatenate(list(decoder('piano.wav')))
Audio(audio.T, rate=44100)
If you're having trouble with audioread you could of course try libsoundfile and see if that works:
import soundfile as sf
y, sr = sf.read('audio.ogg')
Huh! That’s frustrating. Thank you for the detailed tracebacks. I’m guessing this has something to do with Windows executable search vagaries. Can you please try running this in a Python console?
>>> import subprocess >>> subprocess.Popen(['ffmpeg'])My best guess is that your cmd shell has the PATH variable extended to have ffmpeg available, but however your librosa installation somehow does not have the same PATH.
Can you please tell how to solve this issue? I'm currently running on Ubuntu 18LTS and python3.6
I'm running windows 10, installed ffmpeg 4.1 in my conda env, and I get a nobackenderror. Any help?
You will get a NoBackend() exception if ffmpeg fails to read your file (e.g. if you passed a non-audio file by mistake). Double-check that you passed a valid path to an audio file. You can read the output of ffmpeg by putting a breakpoint or print(line) at line 195 of audioread/ffdec.py.
Hello!
This is my first time posting an issue, so please cut me some slack if I'm not following some protocol!
I was trying to use python's
librosapackage on Windows 10 and encountered the following issue.After running
x, _ = librosa.load('data/fma_small/000/000002.mp3', sr = None)I receive this stack trace:--------------------------------------------------------------------------- NoBackendError Traceback (most recent call last) <ipython-input-2-15a8daa0e7fd> in <module>() 1 start, end = 7, 17 2 filename = utils.get_audio_path(AUDIO_DIR, 2) ----> 3 x, sr = librosa.load(filename, sr = None, mono = True) Q:\Program Files\Anaconda3\lib\site-packages\librosa\core\audio.py in load(path, sr, mono, offset, duration, dtype, res_type) 105 106 y = [] --> 107 with audioread.audio_open(os.path.realpath(path)) as input_file: 108 sr_native = input_file.samplerate 109 n_channels = input_file.channels Q:\Program Files\Anaconda3\lib\site-packages\audioread\__init__.py in audio_open(path) 112 113 # All backends failed! --> 114 raise NoBackendError() NoBackendError:Out of curiosity, I also tried to run
audioread.ffdec.FFmpegAudioFile('data/fma_small/000/000002.mp3')(since I know that FFmpeg is installed) but I received:--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in __init__(self, filename, block_size) 126 stderr=subprocess.PIPE, --> 127 stdin=self.devnull, 128 ) Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in popen_multiple(commands, command_args, *args, **kwargs) 88 try: ---> 89 return subprocess.Popen(cmd, *args, **kwargs) 90 except OSError: Q:\Program Files\Anaconda3\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds) 946 errread, errwrite, --> 947 restore_signals, start_new_session) 948 except: Q:\Program Files\Anaconda3\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session) 1223 cwd, -> 1224 startupinfo) 1225 finally: FileNotFoundError: [WinError 2] The system cannot find the file specified During handling of the above exception, another exception occurred: NotInstalledError Traceback (most recent call last) <ipython-input-6-be3b460211e8> in <module>() ----> 1 audioread.ffdec.FFmpegAudioFile(filename) Q:\Program Files\Anaconda3\lib\site-packages\audioread\ffdec.py in __init__(self, filename, block_size) 129 130 except OSError: --> 131 raise NotInstalledError() 132 133 finally: NotInstalledError:Again, I have FFmpeg installed, and
ffmpegworks as a command in cmd.Is anyone able to spot the problem?
Thanks!
Hi, I don't know if it may solve your problem. But for me, some audio files gave this error. So I tried it with different audio files. Most worked, less than 1% of the audio files gave me this error from a huge data set in my project. So check it out. The problem may be due to a specific audio file.
Yes, I get this error if my audio fiels are corrupt or truncated. Note that I also get it sometimes if I load non-corrupt audio files in a multi-threaded context, for example in a pytorch batch data loader which AFAICT is not support, so that is also a possible way of getting this "catch all" error"
I found out that you also get this error even if you have ffmpeg in your path (or in my case, next to your python executable) but with missing DLL's.
ffmpeg tries to run but will error due to missing DLL's and audioread interprets an erroring ffmpeg as "not available".
YES, "fixed" it.
Following this intuition of the
PATHenvironment variable behaving differently from the rest of my system I just copied ffmpeg to the parent folder of the directory and it worked! And then I realized I hardly ever close PyCharm, so I restarted it and it works! This issue also happens with terminals (environment variables don't take effect unless you open a new one). I m having same problems , despite installing ffmpeg. please give me a solution.
my code was: D = [] # Dataset for row in valid_data.itertuples(): y, sr = librosa.load('F:\paper\Bangla ASR\segmented/augmented 1/' + (row.path))
ps = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=52) if ps.shape != (52, 130): continue D.append( (ps, row.classID) )
NoBackendError Traceback (most recent call last)
~.conda\envs\RaffaelEnv\lib\site-packages\librosa\core\audio.py in load(path, sr, mono, offset, duration, dtype, res_type) 117 118 y = [] --> 119 with audioread.audio_open(os.path.realpath(path)) as input_file: 120 sr_native = input_file.samplerate 121 n_channels = input_file.channels
~.conda\envs\RaffaelEnv\lib\site-packages\audioread_init_.py in audio_open(path) 114 115 # All backends failed! --> 116 raise NoBackendError()
NoBackendError:
N.B: I restarted jupyter notebook so many times but still problem exists
try installing ffmpeg with conda ,i was experiencing the same issue and worked for me after installing it with conda ,and not pip,i think its a path problem
Hello, I was also using librosa when the same error occurred when using ffmpeg binary manually added to PATH. Installing av with conda helped resolve the issue.
Anyway, it would be great if we could prevent future issues like these.
Steps to reproduce:
- I downloaded ffmpeg static build from https://johnvansickle.com/ffmpeg/ and added to PATH
$ ffmpeg -version
ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0)
- read file with ffmpeg
audioread.ffdec.FFmpegAudioFile(path)
---------------------------------------------------------------------------
CommunicationError Traceback (most recent call last)
<ipython-input-8-8f1280d75607> in <module>
----> 1 audioread.ffdec.FFmpegAudioFile(path)
~/local_python_libs/audioread/ffdec.py in __init__(self, filename, block_size)
180
181 # Read relevant information from stderr.
--> 182 self._get_info()
183
184 # Start a separate thread to read the rest of the data from
~/local_python_libs/audioread/ffdec.py in _get_info(self)
228 if not line:
229 # EOF and data not found.
--> 230 raise CommunicationError("stream info not found")
231
232 # In Python 3, result of reading from stderr is bytes.
CommunicationError: stream info not found
- original command from librosa
amplitudes, sampling_rate = librosa.load(path, sr=None, mono=False)
What is the issue?
- I use glibc 2.17, but the newest ffmpeg uses gcc 8.3.0 in the static build, that means that it requires minimal version of glibc 2.23
- however the conda package AV has in configuration
--disable-statichere https://github.com/PyAV-Org/PyAV/blob/9ac05d9ac902d71ecb2fe80f04dcae454008378c/scripts/build-deps#L47 - according to https://github.com/FFmpeg/FFmpeg/blob/master/configure it means
Configuration options:
--disable-static do not build static libraries [no]
Proposed solution? Check whether ffmpeg uses static linking if possible. Advice against installing ffmpeg from https://johnvansickle.com/ffmpeg/. Possibly ad github action that checks compatibility of ffmpeg with this library.
That's interesting! I still don't quite see what's wrong with that particular ffmpeg… is it broken altogether? Or is it just not reporting the stream information?
That's interesting! I still don't quite see what's wrong with that particular ffmpeg… is it broken altogether? Or is it just not reporting the stream information?
If I tried to edit locally audioread via Popen, the error is in function available() in ffdec.py. FFmpeg ends with return code 1. However, it runs via the command line. I am not quite sure why it doesn't run via Popen. However, I am pretty sure that the problem is that the binary I used is static build with statically linked libraries such as glibc . I use an older kernel (because I simply have to, it is not my machine, and I don't have root privileges). FFmpeg in conda package AV is built with dynamically linked libraries, so there is no such error with older version glibc.
Quick fix: It would be great to check in audioread for such a type of error and advice for installing ffmpeg from suggested sources or building it without statically linked libraries. I am sure many people run into similar issues, and new people will eventually visit this thread.
Longer fix: Perhaps setup of github actions with simple pytest tests would be reasonable so that in future errors of this nature are prevented. There are many projects using this repository https://github.com/beetbox/audioread/network/dependents, and there is currently many issues with NoBackendError() error https://github.com/search?q=NoBackendError%28%29+audioread&type=issues
I would be happy to elaborate and contribute.
Ah, got it. Thanks for clarifying. I think the thing to do would be to nail down exactly why running the tool on the command line works butPopening it does not. If we can understand that, then maybe we can either (a) fix it, or (b) detect those specific conditions and issue a better error message.
Maybe looking carefully at the stderr/stdout from the process before it exits with status 1 would help shed some light?
I will try to reproduce the error and create a small docker container with these steps.
Okay, so running the ffmpeg without error in cmd was possible due to later installation of ffmpeg with AV. When I try to run it now inside folder with "./" it doesn't run at all.
$ ./ffmpeg
Segmentation fault
Did I install the correct architecture? My architecture is
$ uname -a
Linux XXX 5.4.17-2136.302.7.2.1.el7uek.x86_64 #2 SMP Tue Jan 18 13:44:44 PST 2022 x86_64 x86_64 x86_64 GNU/Linux
and I installed ffmpeg-git-amd64-static.tar.xz.
Again we can look at libc that I am using
$ ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
If we look into build info that John provided it says
Notes: A limitation of statically linking glibc is the loss of DNS resolution. Installing
nscd through your package manager will fix this.
John uses gcc 8.3.0 and looking into https://ftp.gnu.org/gnu/gcc/gcc-8.3.0/ we can see in NEWS
* Support has been added for __builtin_cpu_is() and
__builtin_cpu_supports(), allowing for very fast access to
AT_PLATFORM, AT_HWCAP, and AT_HWCAP2 values. This requires use of
glibc 2.23 or later.
but I am not sure whether this applies to amd64 architecture.
Processor that I am using
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 72
On-line CPU(s) list: 0-71
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 2
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
Stepping: 7
CPU MHz: 2450.644
CPU max MHz: 3900.0000
CPU min MHz: 1000.0000
BogoMIPS: 5200.00
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68
NUMA node1 CPU(s): 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61,65,69
NUMA node2 CPU(s): 2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70
NUMA node3 CPU(s): 3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,67,71
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
so I think I downloaded the correct binary.