Add 'retryAfter' property to 'Too Many Request' Error

[WeeJeWel]

This is a number in the Retry-After header, when the statuscode is 429.

[n1ru4l]

@WeeJeWel Just out of curiosity: How often and under which circumstances do you encounter this statuscode?

[WeeJeWel]

I've created an integration for Homey (www.athom.com) and we have many users, so pretty often already :'(

[n1ru4l]

@WeeJeWel Do you use the users credentials for creating accessTokens or you spotify secret/clientId?

[WeeJeWel]

Both. Users are authenticated using the oauth2 flow, using my id&secret

[n1ru4l]

@WeeJeWel Okay good to know! Thank you for the info.

[gwynnebaer]

I am interested in this as well, but wondering if it should be extended further? For example, as every call uses promises and by definition are asynchronous, I haven't found a way not to immediately overwhelm the rate limit. My use case: Download my playlist metadata including tracks and send IFTTT notifications when a playlist changes.

When I download my playlists, I do not hit the rate limit, but as soon as I start to down the playlist tracks via .getPlaylistTracks(), I get nothing but 429's.

If there is a programmatic way to slow down the rate, then I would code it, but it may be better to just make the Webapi be smarter and gracefully handle 429's for the user? Then there's no guesswork needed on the "slow down" approach (it's usually best practice to gracefully handle the 429's and slow down based on the information returned, than to "guess").

[gwynnebaer]

I am looking at superagent-throttle as a solution. It's working for my needs, and I will clean it up and offer a patch.

[jopek]

@gwynnebaer if i see correctly, using superagent-throttle only allows you to throttle all requests, rather than using the Retry-After header.

@settheset chose to at first automatically retry bases on the Retry-After header but reverted it back to passing the headers to the error object. https://github.com/thelinmichael/spotify-web-api-node/blob/master/src/http-manager.js#L88

Passing the header along with the error object allows one to make use of RxJs' retryWhen operator and supplying the delay operator the Retry-After value.

[kauffecup]

This PR https://github.com/thelinmichael/spotify-web-api-node/pull/237 adds the entire headers object to the error, so consumers can take advantage of the retry-after header. For example, could do something like:

  const doRequest = async (args, retries) => {
    try {
      const response = await spotifyApi.getMySavedTracks(args);
      return response;
    } catch (e) {
      if (retries > 0) {
        console.error(e);
        await asyncTimeout(
          e.headers['retry-after'] ?
            parseInt(e.headers['retry-after']) * 1000 :
            RETRY_INTERVAL
        );
        return doRequest(args, retries - 1);
      }
      throw e;
    }
  };

This creates a wrapper around the request and retries it if the request fails

[wonkoRT]

It would indeed be very useful to add the response headers to the thrown WebapiError. I also need access to "retry-after" but think there might be other uses for the headers as well, if not now maybe in the future.

The rate limit occurs for me when i execute multiple search queries in parallel using await Promise.all.

Took me 5 minutes to patch http-manager.js and webapi-error.js but it would be great if the official npm could include this feature.

For completeness sake, here's the link to the official docs: spotify rate limiting

[omarryhan]

I came up with this and it works like a charm with over 500 request sent at once.

export const callSpotifyWithRetry = async <T>(
  functionCall: () => Promise<T>, retries = 0,
): Promise<T> => {
  try {
    return await functionCall();
  } catch (e) {
    if (retries <= MAX_RETRIES) {
      if (e && e.statusCode === 429) {
        // +1 sec leeway
        const retryAfter = (parseInt(e.headers['retry-after'] as string, 10) + 1) * 1000;
        console.log(`sleeping for: ${retryAfter.toString()}`);
        await new Promise((r) => setTimeout(r, retryAfter));
      }
      return await callSpotifyWithRetry<T>(functionCall, retries + 1);
    } else {
      throw e;
    }
  }
};

results = await callSpotifyWithRetry<SpotifyApi.ArtistObjectFull[]>(async () => {
  const response = await client.getMyTopArtists({
    time_range: 'long_term',
    limit: 50,
  });
  return response.body.items;
});

My current MAX_RETRIES is at 20. I tested once with 10 and it worked just fine. 7 retries however, returned some errors.