ffdec: Dynamic linker errors are misreported as "file not found" errors

[ShaeBrown]

When reading mp3 files I get the following output:

 Traceback (most recent call last):
2017-12-11T06:09:20.901633+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
2017-12-11T06:09:20.901634+00:00 app[web.1]:     response = self.full_dispatch_request()
2017-12-11T06:09:20.901635+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
2017-12-11T06:09:20.901635+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2017-12-11T06:09:20.901640+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
2017-12-11T06:09:20.901641+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2017-12-11T06:09:20.901641+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
2017-12-11T06:09:20.901642+00:00 app[web.1]:     raise value
2017-12-11T06:09:20.901643+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
2017-12-11T06:09:20.901644+00:00 app[web.1]:     rv = self.dispatch_request()
2017-12-11T06:09:20.901644+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
2017-12-11T06:09:20.901645+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2017-12-11T06:09:20.901646+00:00 app[web.1]:   File "/app/flask_app.py", line 27, in display_drum
2017-12-11T06:09:20.901646+00:00 app[web.1]:     song, sr = librosa.core.load(path)
2017-12-11T06:09:20.901647+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/librosa/core/audio.py", line 107, in load
2017-12-11T06:09:20.901648+00:00 app[web.1]:     with audioread.audio_open(os.path.realpath(path)) as input_file:
2017-12-11T06:09:20.901648+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/audioread/__init__.py", line 111, in audio_open
2017-12-11T06:09:20.901649+00:00 app[web.1]:     return ffdec.FFmpegAudioFile(path)
2017-12-11T06:09:20.901650+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/audioread/ffdec.py", line 150, in __init__
2017-12-11T06:09:20.901651+00:00 app[web.1]:     self._get_info()
2017-12-11T06:09:20.901651+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/audioread/ffdec.py", line 206, in _get_info
2017-12-11T06:09:20.901652+00:00 app[web.1]:     raise IOError('file not found')
2017-12-11T06:09:20.901658+00:00 app[web.1]: OSError: file not found

I have the following packages:

audioread==2.1.5
certifi==2017.11.5
chardet==3.0.4
click==6.7
cycler==0.10.0
decorator==4.1.2
Flask==0.12.2
gunicorn==19.7.1
idna==2.6
itsdangerous==0.24
Jinja2==2.10
joblib==0.11
librosa==0.5.1
llvmlite==0.21.0
MarkupSafe==1.0
matplotlib==2.1.0
numba==0.36.1
numpy==1.13.3
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.3
requests==2.18.4
resampy==0.2.0
scikit-learn==0.19.1
scipy==1.0.0
six==1.11.0
tabulate==0.8.2
urllib3==1.22
Werkzeug==0.13

ffmpeg is installed

[sampsyo]

Hi! This error is telling you that the MP3 file you passed in couldn’t be found on the filesystem. Can you check whether the file actually exists? If so, is there a way you can help us reproduce the problem?

[ShaeBrown]

My repo is here: https://github.com/ShaeBrown/csc475/tree/deploy The error is occuring in line 27 in flask_app.py. This call to librosa.load:

    path = os.path.join(app.config['UPLOAD_FOLDER'], filename)
    print(path)
    print(os.path.isfile(path))
    song, sr = librosa.core.load(path)

os.path.isfile(path) prints as true so I'm sure the file is there. The error is occurring in the class FFmpegAudioFile class in the _get_info method. Specifically these lines here:

line = self.proc.stderr.readline()
...
...
if 'no such file' in line:
                raise IOError('file not found')

It only happens with mp3 files and not wav files. This is happening when deployed to heroku but works fine on my local machine.

[sampsyo]

Huh! I'm not sure how to reproduce the problem if it only happens on Heroku. Maybe it can be attributed to a difference in ffmpeg versions? Please update this thread if there's anything I can do to look into it.

[ShaeBrown]

Thanks, I'll try and do some more investigation. Both my computer and Heroku are installing from the same url

http://ca.archive.ubuntu.com/ubuntu/pool/universe/f/ffmpeg/ffmpeg_2.8.11-0ubuntu0.16.04.1_amd64.deb

The same issue seemed to appear here on stackoverflow.

[sampsyo]

Strange! If there's some way to reproduce it in an environment you can control, one way to diagnose the problem might be to print out FFmpeg's output (i.e., the value of line) to see if there's more information about why it can't find the file.

Maybe this is some weird effect of Heroku's filesystem sandboxing?

[ShaeBrown]

The output of line is:

ffmpeg: error while loading shared libraries: libavdevice-ffmpeg.so.56: cannot open shared object file: no such file or directory

Seems that it is an issue with ffmpeg and the filesystem.

Edit: Resolved using this buildpack. Was an issue with the way the heroku apt buildpack was setting LD_LIBRARY_PATH.

[sampsyo]

Great; thank you for investigating!

It looks like there's still something we can improve, however: we could help diagnose this by distinguishing between a case where the audio file isn't found and one of these dynamic linking errors. I'll keep the ticket open regarding that.

[netskink]

fwiw, I ran in to this problem as well. It was very confusing. I thought I had a permission error, or something wrong with my path.

[cheofusi]

Same issue here