Memory leaks from address sanitizer

[kyamagu]

Hi, I was trying running address sanitizer to the unit tests, and found several potential memory leaks. The following is suspicious outputs from the sanitizer:

Indirect leak of 65536 byte(s) in 2 object(s) allocated from:
    #0 0x10b162ab5 in wrap_realloc+0xa5 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45ab5)
    #1 0x126c02fdb in av_realloc_f+0x3d (libavutil.56.dylib:x86_64+0x19fdb)

Indirect leak of 44956 byte(s) in 58 object(s) allocated from:
    #0 0x10b162f33 in wrap_posix_memalign+0xb3 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45f33)
    #1 0x126c02f5e in av_malloc+0x26 (libavutil.56.dylib:x86_64+0x19f5e)

Indirect leak of 1832 byte(s) in 4 object(s) allocated from:
    #0 0x10b162f33 in wrap_posix_memalign+0xb3 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45f33)
    #1 0x126c02f5e in av_malloc+0x26 (libavutil.56.dylib:x86_64+0x19f5e)
    #2 0x124f21c34 in decord::AudioReader::AudioReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int, DLContext, int, bool) audio_reader.cc:25
    #3 0x124f2310b in decord::AudioReader::AudioReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int, DLContext, int, bool) audio_reader.cc:19
    #4 0x124f0e7f7 in decord::runtime::$_0::operator()(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const audio_interface.cc:35
    #5 0x124f0e522 in decltype(std::__1::forward<decord::runtime::$_0&>(fp)(std::__1::forward<decord::runtime::DECORDArgs>(fp0), std::__1::forward<decord::runtime::DECORDRetValue*>(fp0))) std::__1::__invoke<decord::runtime::$_0&, decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*>(decord::runtime::$_0&, decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) type_traits:3694
    #6 0x124f0e486 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<decord::runtime::$_0&, decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*>(decord::runtime::$_0&, decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) __functional_base:348
    #7 0x124f0e416 in std::__1::__function::__alloc_func<decord::runtime::$_0, std::__1::allocator<decord::runtime::$_0>, void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) functional:1558
    #8 0x124f0d135 in std::__1::__function::__func<decord::runtime::$_0, std::__1::allocator<decord::runtime::$_0>, void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) functional:1732
    #9 0x124f2c5ce in std::__1::__function::__value_func<void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) const functional:1885
    #10 0x124f2c520 in std::__1::function<void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const functional:2560
    #11 0x124f2a234 in decord::runtime::PackedFunc::CallPacked(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const packed_func.h:969
    #12 0x124f29f3a in DECORDFuncCall c_runtime_api.cc:233
    #13 0x12293de6e in ffi_call_unix64+0x4e (_ctypes.cpython-38-darwin.so:x86_64+0x2be6e)
    #14 0x7ffee559775f  (<unknown module>)

Indirect leak of 1832 byte(s) in 4 object(s) allocated from:
    #0 0x10b162f33 in wrap_posix_memalign+0xb3 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45f33)
    #1 0x126c02f5e in av_malloc+0x26 (libavutil.56.dylib:x86_64+0x19f5e)
    #2 0x124f21c34 in decord::AudioReader::AudioReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int, DLContext, int, bool) audio_reader.cc:25
    #3 0x124f2310b in decord::AudioReader::AudioReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, int, DLContext, int, bool) audio_reader.cc:19
    #4 0x124f0e7f7 in decord::runtime::$_0::operator()(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const audio_interface.cc:35
    #5 0x124f0e522 in decltype(std::__1::forward<decord::runtime::$_0&>(fp)(std::__1::forward<decord::runtime::DECORDArgs>(fp0), std::__1::forward<decord::runtime::DECORDRetValue*>(fp0))) std::__1::__invoke<decord::runtime::$_0&, decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*>(decord::runtime::$_0&, decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) type_traits:3694
    #6 0x124f0e486 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<decord::runtime::$_0&, decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*>(decord::runtime::$_0&, decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) __functional_base:348
    #7 0x124f0e416 in std::__1::__function::__alloc_func<decord::runtime::$_0, std::__1::allocator<decord::runtime::$_0>, void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) functional:1558
    #8 0x124f0d135 in std::__1::__function::__func<decord::runtime::$_0, std::__1::allocator<decord::runtime::$_0>, void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) functional:1732
    #9 0x124f2c5ce in std::__1::__function::__value_func<void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs&&, decord::runtime::DECORDRetValue*&&) const functional:1885
    #10 0x124f2c520 in std::__1::function<void (decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*)>::operator()(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const functional:2560
    #11 0x124f2a234 in decord::runtime::PackedFunc::CallPacked(decord::runtime::DECORDArgs, decord::runtime::DECORDRetValue*) const packed_func.h:969
    #12 0x124f29f3a in DECORDFuncCall c_runtime_api.cc:233
    #13 0x12293de6e in ffi_call_unix64+0x4e (_ctypes.cpython-38-darwin.so:x86_64+0x2be6e)
    #14 0x7ffee5596c9f  (<unknown module>)

Indirect leak of 538 byte(s) in 36 object(s) allocated from:
    #0 0x10b162820 in wrap_malloc+0xa0 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45820)
    #1 0x126c031b6 in av_strdup+0x30 (libavutil.56.dylib:x86_64+0x1a1b6)

Indirect leak of 306 byte(s) in 2 object(s) allocated from:
    #0 0x10b162ab5 in wrap_realloc+0xa5 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45ab5)
    #1 0x126c0376a in av_fast_realloc+0x52 (libavutil.56.dylib:x86_64+0x1a76a)

Indirect leak of 224 byte(s) in 4 object(s) allocated from:
    #0 0x10b162ab5 in wrap_realloc+0xa5 (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x45ab5)
    #1 0x126bf7f98 in av_dict_set+0x158 (libavutil.56.dylib:x86_64+0xef98)

The full log is attached here: unittests.log

My environment is macOS 11.3 x86_64, and used the following environment to setup and build ASAN-enabled CPython using pyenv and pyenv-alias.

CONFIGURE_OPTS=--with-address-sanitizer \
CC=$(brew --prefix llvm)/bin/clang \
MACOSX_DEPLOYMENT_TARGET=10.14 \
SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk \
VERSION_ALIAS="3.8.9_asan" \
pyenv install 3.8.9
pyenv local 3.8.9_asan
python -mpip install -U pip wheel
python -mpip install nose
cmake -B build -DUSE_CUDA=0 \
    -DCMAKE_C_COMPILER=$(brew --prefix llvm)/bin/clang \
    -DCMAKE_CXX_COMPILER=$(brew --prefix llvm)/bin/clang++ \
    -DCMAKE_CXX_FLAGS="-fsanitize=address -g -O3 -fno-omit-frame-pointer" \
    -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=address"
cmake --build build -j
cd python && \
python setup.py bdist_wheel && \
pip install dist/decord*.whl && \
cd ..
ASAN_OPTIONS=detect_leaks=1 \
python -m nose tests/python/unittests/

[kyamagu]

I think one of the issues is the use of __del__() to destruct reader objects. In Python, __del__ is not suitable for managing resources, and the recommended RAII approach is __enter__, __exit__ pattern.

[dfan]

I observe an increase in memory during the course of training when using the AudioReader during distributed PyTorch training. This doesn't happen when using the VideoReader

[mckinziebrandon]

Has this been resolved? I also seem to be observing a memory leak when using AudioReader.