Every album re-copied on each update

[loafofpiecrust]

I just started using this plugin, and the first time I ran beets alt update ipod after setting things up worked just fine. It took all night, as expected. However, after I added a new album to my library and then ran beets alt update ipod again, it first printed + lines with the new tracks. Then it appeared to print all of my tracks and it looks like it's also going to take hours. I'm guessing it's re-copying over some files each time, which defeats the purpose of this for me coming from using a simple rsync script. Maybe I made a mistake at some point? Or maybe this is an interaction with the extrafiles plugin? Here's my config...

config.yaml

directory: /run/media/snead/icepack/music
library: ~/music/beets-library.db
original_date: yes
format_album: $albumartist - [$original_year/$year] $album
format_item: $artist - $album - $disc.$track. $title
sort_album: albumartist+ year+ album+
per_disc_numbering: yes
import:
  autotag: no
  copy: no
  timid: yes
  languages: en
  incremental: yes
  incremental_skip_later: yes
match:
  preferred:
    countries: ['US', 'GB|UK', 'JP']
    media: ['CD', 'DigitalMedia|File', 'Vinyl']
    original_year: yes
musicbrainz:
  extra_tags: [year]
paths:
  albumtype:soundtrack: Soundtracks/[$realyear] $album/%if{$multidisc,Disc %right{$disc,1}/}$track. $title
  comp: %first{$genre}/[$realyear] $album/%if{$multidisc,Disc %right{$disc,1}%if{$grouping,. $grouping}/}$track. $title
  albumtype:single: %the{$albumartist}/Singles/%if{$subartist,[$subartist]}[$realyear] $album/$track. $title
  default: %the{$albumartist}/[$realyear]%if{$subartist,[$subartist]} $album%ifdef{atypes, $atypes}/%if{$multidisc,Disc %right{$disc,1}%if{$grouping,. $grouping}/}$track. $title
item_fields:
  multidisc: 1 if disctotal > 1 else 0
  # MF Doom is something special, with all of his alternate names
  subartist: albumartist_credit if albumartist.lower() != albumartist_credit.lower() else ''
  realyear: original_year if original_year != 0 and original_year < year else year
plugins: the badfiles albumtypes missing rewrite fromfilename inline extrafiles fetchart alternatives convert duplicates
# Automatically resample all hi-fi FLACs down to CD quality on import
convert:
  auto: no
  embed: no
  max_bitrate: 1411
  never_convert_lossy_files: yes
  format: flac
  formats:
    flac: ffmpeg -y -i $source -acodec flac -af aresample=osf=s16:dither_method=triangular:resampler=soxr:out_sample_rate=44100 $dest
fetchart:
  auto: no
  cautious: yes
  cover_names: cover front folder album art
  minwidth: 1000
  enforce_ratio: 2%
  sources: filesystem coverart itunes
  deinterlace: yes
extrafiles:
  patterns:
    images: '*.(jpg|jpeg|png)'
albumtypes:
  types:
    - ep: 'EP'
    - single: 'Single'
    - live: 'Live'
    - compilation: 'Comp'
  ignore_va: compilation
  bracket: '()'
rewrite:
  albumartist DANGERDOOM: MF DOOM
  albumartist Viktor Vaughn: MF DOOM
  albumartist The Jimi Hendrix Experience: Jimi Hendrix
  albumartist Mychael Danna, DeVotchKa: DeVotchKa
  albumartist Adrian Orange: Thanksgiving
  albumartist The Velvet Underground & Nico: The Velvet Underground
  albumartist kidkanevil & Daisuke Tanabe: Daisuke Tanabe
  albumartist David Byrne & St. Vincent: David Byrne
  albumartist filous & Daði Freyr: Daði Freyr
  albumartist J\.U\.F: Gogol Bordello
  albumartist Gorillaz feat.*: Gorillaz
  albumartist Spacemonkeyz vs. Gorillaz: Gorillaz
  albumartist Justin Timberlake & .*: Justin Timberlake
  albumartist The Management: MGMT
  albumartist Urban Thermo Dynamics: Mos Def
  albumartist The Flaming Lips .*: The Flaming Lips
  albumartist Prince and The Revolution: Prince
alternatives:
  ipod:
    directory: /run/media/snead/SHELBY_S IP/music-beets
    query: "onplayer:true"
    removable: true
replace:
  '[\?]$': ''
  '[\\/]': "-"
  '^\.': _
  '<\|': '◀'
  '\|>': '▶'
  '[\x00-\x1f]': _
  ':': ';'
  '"': "'"
  '[<>\?\*\|]': _
  '\.$': ''
  '\s+$': ''
  '^\s+': ''
  '^-': _
  '‐': '-'
duplicates:
  album: yes
  keys: [albumartist, album, year]

[geigerzaehler]

Hi @loafofpiecrust, thanks for reporting this. The config looks good to me and the second update should not try to copy the files again. You’re probably right that one of the plugins (or maybe some custom configuration) is causing the problem.

Could you try to isolate the issue by disabling plugins and removing custom configuration? Then I can try to identify the problem with a simpler configuration.

[loafofpiecrust]

Thanks for the quick follow-up! I removed most of my plugins, reduced the config a little. I tested on my local BTRFS filesystem and things worked as expected. Then I tested again with my iPod which has a FAT32 filesystem and the issue recurred, but only with certain files. At first I added some FLACs and it worked fine. Then I added a few more albums and the issue happened with those files (MP3 and FLAC). Must be related to FAT32 filesystem.

[loafofpiecrust]

Okay sorry I forgot something. It seems like I made a mistake. The issue is that when I set my onplayer=true tag I used -W to avoid writing the tag to disk. This is what caused those specific files to be copied over during every single update. I just tested removing that tag entirely, then re-writing it without -W and I got the right behavior with the same files. I didn't want this tag in my files, but so it goes! This might be worth adding to the docs for this plugin though. Interestingly, using -W was still only an issue on FAT32.

EDIT: I've since tested with my whole library again after letting it write onplayer to the files, and still running into the same issue.

[geigerzaehler]

Thanks for the update. I think I’ve found the problem and it has to do with modification times on FAT32 only having second resolution. I’ve pushed a patch and you can try it out to see if that fixes your issue:

pip install --user git+https://github.com/geigerzaehler/beets-alternatives.git@fat32-mtime

[geigerzaehler]

@loafofpiecrust, did you have a chance to try out my branch and see if it fixes the problem?

[loafofpiecrust]

I figured out how to build your branch to include it in my beets build, and tested it out. It does not appear to fix my issue, and it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod. The destination files have a brand new mtime way later than the source mtime, yet it keeps copying them over. It must have something to do with the destination path itself?

[loafofpiecrust]

@geigerzaehler There doesn't seem to be a pattern to which files are getting copied every time. Some are whole albums, some individual tracks. Perhaps more non-alphanumeric characters? I'm not sure. I do have several extra replace entries in my config to replace problematic characters... Including a few here in case that's helpful.

+/run/media/snead/SHELBY_S IP/Music/4 Non Blondes/[1992] Bigger, Better, Faster, More!/03. What's Up¿.flac
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2007] Collaborations (Comp)/01. Ayo Technology.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/01. That's What Love Can Do.mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/03. Beautiful (Valentin club mix).mp3
+/run/media/snead/SHELBY_S IP/Music/Justin Timberlake/[2003] Justin & Christina (EP)(Comp)/05. Fighter (Hellraiser remix).mp3
+/run/media/snead/SHELBY_S IP/Music/Anamanaguchi/[2006] Power Supply (EP)/06. Flora⁄Fauna.mp3
+/run/media/snead/SHELBY_S IP/Music/Anderson .Paak/[2016] Malibu/04. The Season ⁄ Carry Me.mp3
+/run/media/snead/SHELBY_S IP/Music/Nujabes/[2003][Various Artists] Hydeout Productions; First Collection (Comp)/03. Strive.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/01. The Narrator.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/02. Tidal Wave.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/03. High Tide.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/04. Green Machine.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/05. Winter Must Be Cold.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/06. She's Just Like Me ⁄ Taking Time.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/07. Glowworm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/08. Dots 1-2-3.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/09. Lucky Charm.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/10. Innerspace.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/11. Show the World.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/12. Love You Alice ⁄ D.mp3
+/run/media/snead/SHELBY_S IP/Music/Apples in Stereo, The/[1995] Fun Trick Noisemaker/13. Pine Away.mp3
+/run/media/snead/SHELBY_S IP/Music/Arcade Fire/[2007] Neon Bible/05. Black Wave⁄Bad Vibrations.flac
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/01. Do I Wanna Know¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/02. R U Mine¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Arctic Monkeys/[2013] AM/09. Why'd You Only Call Me When You're High¿.mp3
+/run/media/snead/SHELBY_S IP/Music/Grizzly Bear/[2007] Friend (EP)/09. Knife.mp3

[loafofpiecrust]

I had to make a few more changes to use existing untracked destination files imported by my rsync script on this branch (EDIT: I also did remove art embedding since I use cover files). It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary. Much more robust to manual renames/moves. As an aside, it does seem quite slow to finish when there are zero updates needed (4 minutes on my library!).

[geigerzaehler]

it looks like your code change would avoid unnecessary WRITE actions. Actually, I'm getting unnecessary ADD actions on the same files every time I run beet alt update ipod.

Yes, you’re right! I didn’t realize that the problem was that files were added.

It seems to work fine now with yours and my changes together, updating metadata on my destination when necessary.

I took a look at your branch and I couldn’t quite figure out what the code you changed does. But to me it looks like it breaks things because we’re effectively doing self.set_path(item, self.destination(item)) which mixes library paths with paths in the alt collection.

I think it works for you at the moment because your basically skipping adding a file when the path is not found which seems to suggest that the paths don’t actually exist or os.path.lexists(syspath(path)) behaves strangely on fat32.

One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.

[wisp3rwind]

I haven't been following this issue up to now; and I'm not sure that I understand correctly which steps are required to trigger the extraneous ADD actions. Maybe you could clarify in terms of the actual beet commands? From what I gather, it should be something like:

• configure an alternative ipod on a FAT32 filesystem (required, other filesystems don't exhibit the issue) which includes tracks according to the onplayer:true query
• beet modify -W <some query> onplayer=true (is the -W required? you used it initially, but in the end, the issue occurred without, right?)
• beet alt update ipod
• beet import <some new music>
• beet modify -W <query for new music> onplayer=true
• beet alt update ipod (should add only the new music, but in fact also adds the old tracks again)

Some questions to ponder:

• Does the same issue occur on a plain FAT32 thumb drive (or a FAT32 loop device) (i.e. is there a chance that the iPod does something weird with mtimes or file name normalization? Although mtimes shouldn't be the issue, since they would lead to WRITEs.)
• Could we add a FAT32 test case to CI by creating a FAT32 partition on a loop device and running the tests there?
• Could you repeat the steps and report the files' mtimes after each of them?
• As @geigerzaehler said:

  One thing you could do is debug the code in matched_item_actions to figure out why the files do not exist.