ArtResizer holds file handles for every album art created when using the convert plugin

[afontenot]

Problem

I was trying to convert a large number of files with beet convert -d <dest>, and eventually got an error:

error: convert: couldn't invoke 'ffmpeg -i <SOURCEFILE> -y -vn -acodec libopus -ab 96k <DESTFILE>': [Errno 24] Too many open files

So far I'm on my third invocation of beet convert (existing files get skipped, so it eventually completes). Looking at lsof, it seems that many hundreds of file handles are kept open in /tmp/beets/util_artresizer/, which can run up against default OS limits pretty easily.

I don't think the art resizer should need to keep file handles open, it could just keep the path to the file in case it is needed again.

Setup

• OS: Ubuntu 22.04.4
• Python version: 3.10.12
• beets version: 2.2.0
• Turning off plugins made problem go away (yes/no): n/a (problem affects a plugin)

My configuration (output of beet config) is:

directory: /mnt/data/audio
# --------------- Main ---------------

library: ~/.config/beets/music.blb
pluginpath: ~/.local/share/beets/plugins

# --------------- Plugins ---------------

plugins: convert copyartifacts fetchart inline permissions replaygain zero info mbsync
medium_rec_thresh: -1.0

import:
    detail: yes
    move: yes
    timid: yes
    searchlimit: 8
copyartifacts:
    extensions: .cue .log
    print_ignored: no
zero:
    fields: albumartist_credit albumstatus artist_credit acoustid_id acoustid_fingerprint
    auto: yes
    keep_fields: []
    update_database: no
art_filename: folder

# --------------- Tagging ---------------

per_disc_numbering: yes
incremental: yes
artist_credit: yes
paths:
    albumtype:soundtrack: soundtracks/$authorcredit/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumtype:single: singles/$albumartist/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumgenre:soundtrack: soundtracks/$authorcredit/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumgenre:classical: composers/$authorcredit/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumgenre:instrumental: composers/$authorcredit/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumgenre:experimental: experimental/$albumartist/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    albumgenre:jazz: jazz/$albumartist/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    comp: music/$albumartist/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title
    default: music/$albumartist/$album ($year)%aunique{}/%if{$multidisc,Disc $disc/}$track - $title

aunique:
    keys: albumartist album year
    disambiguators: albummediatype albummedia label catalognum
album_fields:
    albummediatype: "m_0 = items[0].get(\"mediatype\")\nif m_0 and all(m_0 == item.get(\"mediatype\") for item in items[1:]):\n    return m_0\nreturn \"\"\n"
    albummedia: "m_0 = items[0].get(\"media\")\nif m_0 and all(m_0 == item.get(\"media\") for item in items[1:]):\n    return m_0\nreturn \"\"\n"
    albumgenre: "try:\n    return classification\nexcept NameError:\n    pass\ntry:\n    return genre.lower()\nexcept NameError:\n    return \"\"\n"
    authorcredit: "c_0 = items[0].get(\"composer\")\nif c_0 and all(c_0 == item.get(\"composer\") for item in items[1:]):\n    return c_0\na_0 = items[0].get(\"artist\")\nif a_0 and all(a_0 == item.get(\"artist\") for item in items[1:]):\n    return a_0\nreturn albumartist\n"
item_fields:
    multidisc: 1 if disctotal > 1 else 0

# --------------- Paths ---------------

path_sep_replace: "\u29F8"
replace:
    \\: "\u29F8"
    ^\.: ''
    '[\x00-\x1f]': ''
    \.$: ''
    \s+$: ''
convert:
    format: opus
    never_convert_lossy_files: yes
    album_art_maxwidth: 1080
    dest:
    pretend: no
    link: no
    hardlink: no
    threads: 8
    id3v23: inherit
    formats:
        aac:
            command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
            extension: m4a
        alac:
            command: ffmpeg -i $source -y -vn -acodec alac $dest
            extension: m4a
        flac: ffmpeg -i $source -y -vn -acodec flac $dest
        mp3: ffmpeg -i $source -y -vn -aq 2 $dest
        opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
        ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
        wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
    max_bitrate:
    auto: no
    auto_keep: no
    tmpdir:
    quiet: no
    embed: yes
    paths: {}
    no_convert: ''
    copy_album_art: no
    delete_originals: no
    playlist:
replaygain:
    backend: ffmpeg
    overwrite: no
    auto: yes
    threads: 8
    parallel_on_import: no
    per_disc: no
    peak: 'true'
    targetlevel: 89
    r128: [Opus]
    r128_targetlevel: 84
pathfields: {}
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    quality: 0
    max_filesize: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    sources:
    - filesystem
    - coverart
    - itunes
    - amazon
    - albumart
    - cover_art_url
    store_source: no
    high_resolution: no
    deinterlace: no
    cover_format:
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
permissions:
    file: '644'
    dir: '755'

[wisp3rwind]

As noted in #5746, it might helpful to know in more detail what beets is doing before the crash. (In particular, which backend the art resizer is using, but also which operations it performs and whether there are any errors. Not sure whether the log will really contain all of this, but at least some of it.)

Could you provide a verbose log of the operation, i.e. beet -vv convert ...?

[afontenot]

I'll try to rerun it soon (although it's a pain and will take time because by the nature of the issue, I effectively have to reconvert most of my collection to encounter the bug). I do know the artresizer method is PIL.

Sample log portion:

artresizer: method is PIL
convert: image size: (1500, 1500)
convert: embedding album art from /mnt/data/audio/music/100 gecs/10,000 gecs (2023)/folder.jpg
convert: Resizing album art to 1080 pixels wide and encoding at quality               level 0
artresizer: PIL resizing /mnt/data/audio/music/100 gecs/10,000 gecs (2023)/folder.jpg to /tmp/beets/util_artresizer/resize_PIL_8nm0c1g0.jpg

[afontenot]

Okay, I reran it. There's nothing else in the log that seems to be relevant, it's just the "resizing album art" lines I copied above and eventually the "couldn't invoke" error I already reproduced above. I'll upload the log somewhere if you think you need it.

What's probably more useful: I tracked beets' open files while running beet convert. It died with between 1000 and 1100 files open, and at this time, more than 90% of the open files were in /tmp/beets/util_artresizer/

I then increased the soft limit ulimit -n from 1024 to 8192, deleted my output directory, and reran beet convert. It completed with no errors. So the issue here seems pretty straightforward, it's just a matter of figuring out why the PIL resizer (I thought it was the only one tbh) holds open file handles.

[afontenot]

I updated to 2.3.1 and lost the ability to reproduce this. The number of open file handles stays constant (and low) now. I believe this was fixed as part of some refactoring to the album art code in 2.3.0, but (though I tried) I wasn't able to find a specific commit that looked relevant.

[wisp3rwind]

Nice; feel free to re-open if the issue recurs!