Spotify's auto-generated playlists are cached forever

[Qluxzz]

Describe the bug Spotify has auto generated playlists, such as Discover Weekly, Release Radar and etc. In the case of Discover Weekly, it's updated every Monday.

To Reproduce Steps to reproduce the behavior:

1. Add the Discover Weekly playlist to your playlists
2. Wait until Monday when the playlist is updated
3. Go to the playlist in ncspot, and compare it with in the official client
4. The playlists will not be the same

Expected behavior The playlist should be the same as seen in the official client

System (please complete the following information):

• OS: Fedora
• Terminal: alacritty
• Version: 0.5.0
• Installed from: COPR

Additional context The updated playlist and the cached playlist have the same snapshot id, so it's not fetched when loading the cached info, since we compare snapshot ids in order to determinate if we should update the playlist.

The question might be why they still have the same snapshot id even after clearly being updated?

https://github.com/hrkfdn/ncspot/blob/f2bcfcaa605ccd38be15d87aab15dd1606cb80a6/src/library.rs#L101-L113

[gmontanola]

@Qluxzz about the workaround "It can be resolved by running the update command manually"

You mean using Shift+U? Or is there another way that I'm not aware (because this is not working here)

But removing ~/.cache/ncspot/playlists.db does work.

[Qluxzz]

@gmontanola seems like I was mistaken, the update command does exactly the same as Shift+U and the same as on startup when we update the library.

I was talking about the vim style command prompt which opens by default when pressing :

So the only solution is to remove the ~/.cache/ncspot/playlists.db file.

[hrkfdn]

Hrrmm.. odd. Mine seem to be updating fine, so I'm guessing the snapshot id updates for me. Question is why it doesn't for you guys.

[Qluxzz]

Here's the result from when I fetch GET https://api.spotify.com/v1/users/{user_id}/playlists, the same as ncspot does refreshing the cached playlists.

{
  "href": "https://api.spotify.com/v1/users/qluxzz/playlists?offset=7&limit=1",
  "items": [
    {
      "collaborative": false,
      "description": "Your weekly mixtape of fresh music. Enjoy new music and deep cuts picked for you. Updates every Monday.",
      "external_urls": {
        "spotify": "https://open.spotify.com/playlist/37i9dQZEVXcLkbwtizsRxa"
      },
      "href": "https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa",
      "id": "37i9dQZEVXcLkbwtizsRxa",
      "images": [
        {
          "height": null,
          "url": "https://newjams-images.scdn.co/v3/discover-weekly/SxA_IlWkvtKuDBHNo_L8e03kkVwxL9SbhQFCj23a0EClGTt-nxPSun-vmz0bIfgIpehVXoQBI7Gheq5A4Jp04K76yyMfT2qXJVSIVpENsD0=/MjA6ODA6NzBUMzAtNDAtMQ==/default",
          "width": null
        }
      ],
      "name": "Discover Weekly",
      "owner": {
        "display_name": "Spotify",
        "external_urls": {
          "spotify": "https://open.spotify.com/user/spotify"
        },
        "href": "https://api.spotify.com/v1/users/spotify",
        "id": "spotify",
        "type": "user",
        "uri": "spotify:user:spotify"
      },
      "primary_color": null,
      "public": false,
      "snapshot_id": "MCwwMDAwMDAwMGM5NzM0NmMxOWMwNmE3YzgyYjc1MzIyZjUyZGU5OWU1",
      "tracks": {
        "href": "https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa/tracks",
        "total": 30
      },
      "type": "playlist",
      "uri": "spotify:playlist:37i9dQZEVXcLkbwtizsRxa"
    }
  ],
  "limit": 1,
  "next": "https://api.spotify.com/v1/users/qluxzz/playlists?offset=8&limit=1",
  "offset": 7,
  "previous": "https://api.spotify.com/v1/users/qluxzz/playlists?offset=6&limit=1",
  "total": 28
}

The snapshot id in the response is: MCwwMDAwMDAwMGM5NzM0NmMxOWMwNmE3YzgyYjc1MzIyZjUyZGU5OWU1 The snapshot id in playlists.db is: MCwwMDAwMDAwMGM5NzM0NmMxOWMwNmE3YzgyYjc1MzIyZjUyZGU5OWU1

Fetching the tracks of the playlist, this is returned:

{
  "collaborative": false,
  "description": "Your weekly mixtape of fresh music. Enjoy new music and deep cuts picked for you. Updates every Monday.",
  "external_urls": {
    "spotify": "https://open.spotify.com/playlist/37i9dQZEVXcLkbwtizsRxa"
  },
  "followers": {
    "href": null,
    "total": 1
  },
  "href": "https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa",
  "id": "37i9dQZEVXcLkbwtizsRxa",
  "images": [
    {
      "height": null,
      "url": "https://newjams-images.scdn.co/v3/discover-weekly/rus4kJZOvPKKMvXsVTaDGd-lLbaxGjd25mHlYvufFWReDniokQvNtimdJviVEC9qywhYo-bxXlnZLkfjM7cfqy5Ipb8PjlAcQ_8YmK1oyJc=/MjA6MDM6NzBUMzAtNDAtMQ==/default",
      "width": null
    }
  ],
  "name": "Discover Weekly",
  "owner": {
    "display_name": "Spotify",
    "external_urls": {
      "spotify": "https://open.spotify.com/user/spotify"
    },
    "href": "https://api.spotify.com/v1/users/spotify",
    "id": "spotify",
    "type": "user",
    "uri": "spotify:user:spotify"
  },
  "primary_color": null,
  "public": false,
  "snapshot_id": "MCwwMDAwMDAwMDE1YzRmMjdmMWI2YTk1NzJjNDE1N2EzMGUxMGMzNWJj",
  "tracks": {

But now the snapshot id is changed: MCwwMDAwMDAwMDE1YzRmMjdmMWI2YTk1NzJjNDE1N2EzMGUxMGMzNWJj

Seems to be a bug in the GET https://api.spotify.com/v1/users/{user_id}/playlists endpoint but then it's weird it doesn't affect you @hrkfdn.

[Qluxzz]

When I started ncspot and navigated to the "Discover Weekly" playlist it is now updated, I'm just guessing but seems like there is some cache invalidation problem with the endpoints, and by me fetching the playlist's tracks, the cache was purged for the GET https://api.spotify.com/v1/users/{user_id}/playlists endpoint maybe

[Qluxzz]

I've opened an issue on Spotify for clarification https://community.spotify.com/t5/Spotify-for-Developers/snapshot-id-not-updated-for-https-api-spotify-com-v1-users-user/m-p/5172191#M2203

[hrkfdn]

[2021-03-22][20:55:12] [ncspot::library] [DEBUG] Release Radar, local id: NDQ4LDAwMDAwMDAwN2UxMDc4YWQ1ZDNkNjk0OWJmNmI5MjIwODM3MzdmYWM=, remote id NDQsYTI4YjhkNWZiYzM2NGI3NGVmMjU5NDdlOTA2NWMzMzQ0MjMwZjY0YQ==
[2021-03-22][20:55:12] [ncspot::library] [DEBUG] Discover Weekly, local id: MCwwMDAwMDAwMDFjZTdlOTEzNjIzOWMwOWJlMTk1MGFmZDA5ZDZkNTZj, remote id NDQsYTI4YjhkNWZiYzM2NGI3NGVmMjU5NDdlOTA2NWMzMzQ0MjMwZjY0YQ==

Yeah it seems fine here, weird.

Thanks for opening the issue.

[Bettehem]

For me the playlists get updated normally too. I have only tested with Discover Weekly though.

[Qluxzz]

New week, still cached. Trying to figure out why it works for some users but not myself and @gmontanola.

Since fetching the tracks for the playlists updates the snapshot_id, but only fetching the playlist does not.

I ran ncspot with the -d command and tried to diff the ids as you did @hrkfdn, but then it worked and it updated the playlist correclty.

Here's the only thing I can find in the log about it: [ncspot::library] [INFO] updating playlist Discover Weekly (index: 7)

[Qluxzz]

New week so I can try again.

The cached snapshot_id: MCwwMDAwMDAwMDI3NTdlMmQzOTEyNDc2NDI2ODU2NWQwMjZmNWY5ZTc3

Endpoint: https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa Returns snapshot_id: MCwwMDAwMDAwMDgxYTE0NTAxNDVhYjBlYTQ3NWQ4MTg5OGE5OGYwMWI3

Endpoint: https://api.spotify.com/v1/me/playlists Returns snapshot_id: MCwwMDAwMDAwMDI3NTdlMmQzOTEyNDc2NDI2ODU2NWQwMjZmNWY5ZTc3

Calling https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa after calling https://api.spotify.com/v1/me/playlists still returns a different snapshot_id, so it's starting to look like a cache invalidation problem at Spotifys Web API.

Calling https://api.spotify.com/v1/playlists/37i9dQZEVXcLkbwtizsRxa/tracks and then calling https://api.spotify.com/v1/me/playlists yields yet another snapshot_id: MCwwMDAwMDAwMDJiNGMzMGYwMDEyMDRkNzNmMWVhZjY4MmVjZWIzZjA0

But now the cache for https://api.spotify.com/v1/me/playlists is successfully cleared and the playlist will be updated when ncspot is started.

[xpayn]

Maybe @Bettehem and @hrkfdn are using other clients that poll the tracks of the playlist and thus invalidate the cache, but if you only use ncspot the cache is never invalidated.

[Qluxzz]

@xpayn the weird thing is that I use the official Android client as well, but I doubt that Spotify use the web API within their own applications, otherwise it feels like they would have found the cache problem by now.

[xpayn]

@Qluxzz yep that sounds right. Anyway, this morning I opened ncspot and got last week playlist, so I tried to load it from the iphone app and restart ncspot, but it's still the old one. So my hypothesis is false.

EDIT: after a refresh I got the new songs, but I didn't try it before launching the ios app, so no conclusion can be drawn

[Bettehem]

I experienced this issue for the first time now too. I didn't open the mobile app or the official desktop app before this. However, once I did, my discover weekly list was updated with new songs in the official client, while ncsspot's list still isn't.

EDIT: 3 hours later, I'm checking my list again and it is now updated in ncspot too.

[FekketCantenel]

I also just noticed this issue today with the Discover Weekly playlist. When I use SHIFT+U to update, the [U] appears for about a second and then disappears without updating anything.

Since I installed ncspot through snap, my path to the temp file is ~/snap/ncspot/89/.cache/ncspot/playlists.db. (I also have the same path but for /85/, ignored it.) Renaming that file and using the :reload command brought in the new Discover Weekly playlist.

Edit: I got confused and renamed the file in the /85/ version. No new one was created, and the corresponding file in /89/ has today's modified date.

[ghost]

In a similar fashion is it expected behavior to have a 7gb cache after 1 month of use? I know I can just remove it manually and it's not that much of a problem but I was surprised.

[hrkfdn]

@spouradrix That's local audio file caching. There is no cleanup strategy for the local cache yet, but the next librespot release will allow ncspot to have a size limit for the local cache: https://github.com/librespot-org/librespot/pull/675

[szaffarano]

Hi, the feature is already merged