[BUG]: Exceeding daily throttle limit unnecessarily

[samborambo305]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

I have roughly the following code (simplified of course):

const allPayments = [];
let more = true;
while (more) {
  const payments = await etsy.getShopPaymentAccountLedgerEntries(
    minDate,
    endDate,
    offset
  );
  await sleep(100);

  allPayments.push(...payments);

  if (payments.length === 100) {
    offset += 100;
  } else {
    more = false
  }
}



for await (const data of allPayments) {
  if (ledgerType === 'PAYMENT_GROSS') {
    const payment = await etsy.getPaymentAccountLedgerEntryPayments(entryId);
    const shopReceipt = await etsy.getShopReceipt(payment[0].receipt_id);
    await sleep(200);

    // do stuff
  }
}

I am running this code with a min date of 1/1/22 and end date of 10/12/22. Before this code completes, I get an error saying that i "Exceeded daily rate limit".

Per the docs, there are 10k requests per day. It seems impossible that this code would yield this (especially after only a few minutes of running).

Steps to reproduce

Run the above code^

It is not ready to run as I don't want to put all my production code on the internet, but the function names are provided and match the API function names

Expected behavior

I am able to run this code to completion without it complaining about throttle limits.

Additional context

No response

[samborambo305]

@etsyachristensen

[etsyachristensen]

@samborambo305 Are you sure you're not sending too many requests per second? We have both a daily limit and a per second limit. These limits are global as well as per token.

Can you provide the headers you are getting back when this happens? (just redact your token credential info)

[samborambo305]

After monitoring the headers, I am realizing that the 10k limit is actually being violated. Calling getPaymentAccountLedgerEntryPayments and getShopReceipt for each item in all of my payouts from January through today actually is putting me over this threshold. Surely there is another way to do this that won’t require so many API calls? Or perhaps the threshold is too low?

[etsyachristensen]

No that is the standard threshold. Most apps don't request that much data all at one time. I'd recommend narrowing your time range parameters to smaller date ranges and spread it out over multiple days. You can try to request an increase in your limit by contacting the developer support at [email protected]. However, it is unlikely to be approved as this is rather out of the norm.

[samborambo305]

I'm not understanding why this isn't a normal operation to perform. I am trying to get all the ledger entry payments and shop receipts for all my payouts, year-to-date. I want to be able to display this to users in my accounting software and this is the onboarding step. Is there a more efficient way to do this? the pseudocode i provided initially is showing the general approach i am taking

I need to run things this way to differentiate orders that collect marketplace facilitator taxes from those that don't since this is relevant to bookkeeping.

[samborambo305]

Are there alternative ways to achieve the same thing?

[etsyachristensen]

@samborambo305 The issue isn't that the rates are too low or that what you are trying to get to is out of the ordinary. The volume is. Trying to get year to date is likely going to max you out. You'd be better off breaking it down into chunks of time. For instance request one month at a time or one week at a time, depending on how many entries you have. Then repeat, but add some wait time in between. Your initial call gets all the entries, but then for each entry you're calling to payment and receipt endpoints. So if there are a lot of entries in the year, like 5,000 or more, then you'll surely hit the max in a day. Likely as not, you'll also hit the per-second limit as well because the script will fire off pretty fast.

I recommend that your application be designed to fetch data in smaller chunks and do so on a regular basis so that it's not having to go back very far. Maybe set it up to run daily or weekly, depending on the shop's sales volume.