NREL
Lots of random "Data processing failure." from NSRDB API
My calls to the NSRDB API have started failing quite frequently, returning "Data processing failure." And it seems random; if I make the same request repeatedly, it might fail 1/3 three times. I've rarely seen this error before, and nothing has changed on my end. I'm using these endpoints:
- https://developer.nrel.gov/api/nsrdb/v2/solar/psm3-download.csv
- https://developer.nrel.gov/api/nsrdb/v2/solar/himawari-tmy-download.csv
I suspect this has to do with rate limiting, as I'm only encountering issues when running multiple requests in a row. Have rate limiting policies changed? The policies for CSV downloads are listed as:
5000 requests per day 1 request per second 20 requests in process at once
I could be bumping into the "1 request per second" rule, but this is surprising, as I've been running the same job for several years now, and it only became an issue today.
Thanks!
Today my requests are going through just fine! Did anything change? Were y'all having infra issues?
if I make the same request repeatedly, it might fail 1/3 three times
Once upon a time I experienced a similar issue, but with 50% of requests failing. Turned out to be some issue behind the API's load balancer where one (out of two total) servers was erroring. Maybe there are three servers now :)
Long time no see @kandersolar ! That sounds likely.
@caseyzak24 also suggested it might be this:
In addition, the service has a fail-safe mechanism to prevent significant performance decreases that can be caused by unexpectedly high usage of the service. This limit will cause the service to stop accepting requests when the queue reaches a point where additional requests will significantly lower server performance. When this limit is hit, the service will error with a message describing that the request queue is full.
I didn't get a message about a queue being full, but 🤷
Hey folks, sorry I missed this one and am slow to respond!
Yes, our IT department has reported network bandwidth issues that required network throttling of our services, which are operating at less than half capacity as a result. I didn't adjust the rate limits given that this is a "short" term issue. However the result is that our services are prone to start failing intermittently as soon as load ramps up. I am led to believe that the network issue will be resolved this week and I will then spin our services back up to full operational capacity. I'll come back and close this ticket when that actually happens.
OK, we're back to full capacity! I'm going to close this. Let me know if you have continued access issues.
Seems like this issue is happening again?
Yes, we're having some server issues resulting in intermittent failures. I'm working on a solution and I'll let you know when we're back to full capacity.
Seems like this issue is happening again?
Yes, we're having some server issues resulting in intermittent failures. I'm working on a solution and I'll let you know when we're back to full capacity.
OK, we had just a couple of specific servers with issues this time. I've taken them out of the loop and we're back in action at only slight less than full capacity.
Is this happening again? I'm getting: status":400,"errors":["Data processing failure."] I have tested so many variants of my API call, trying to simplify the parameters or test different parameters, but I always get it.
I get this error for every call. I've already ran: print(f"Remaining: {response.headers.get('X-RateLimit-Remaining')}") print(f"Limit: {response.headers.get('X-RateLimit-Limit')}") print(f"Reset: {response.headers.get('X-RateLimit-Reset')}") and supposedly, I have like 4980 remaining on my limit.
Here is my current call:
https://developer.nrel.gov/api/nsrdb/v2/solar/nsrdb-GOES-tmy-v4-0-0-download.csv?api_key=redacted&wkt=POINT%28-122.330062+47.6038321%29&attributes=dni%2Cdhi%2Cghi%2Csolar_zenith_angle%2Cair_temperature%2Ccloud_type&names=tdy-2023&interval=60&utc=false&email=fake.email%40yahoo.com&leap_day=false
Yes, I'm just deploying a fix now. Believe it or not a completely unrelated issue to the first couple of times. There may be some intermittent failures for the next hour or so while the fix propagates out to our entire infrastructure fleet.
The bugfix is deployed on our side. Has this resolved the issues you were seeing?
Sorry for not replying to this before. Any chance you are having this issue again? @PjEdwards I just started getting this show up in my logs again.
https://developer.nrel.gov/api/nsrdb/v2/solar/psm3-tmy-download.csv?names=tmy&wkt=POINT%28-121.7738175+37.6766133%29&interval=60&api_key=<KEY>&email=<EMAIL>&utc=false
{"inputs":{"body":{},"params":{},"query":{"names":"tmy","wkt":"POINT(-121.7738175 37.6766133)","interval":"60","email":"<EMAIL>","utc":"false"}},"metadata":{"version":"2.0.0"},"status":400,"errors":["Data processing failure."]}
Getting lots of "400: Data processing failure" since yesterday. Outage?
Let me take a look. We did have a deployment snafu yesterday, but that only resulted in about 10 minutes of downtime for all the endpoints.
Thank you for looking into this. For reference We have seen an outage since yesterday.
OK, a fix has been deployed. Please test things out and close if you find everything is back to normal.
still not working for us and we are debugging and trying to see why. will update you soon. For referece we have been testing since your last message.
still not working for us and we are debugging and trying to see why. will update you soon. For referece we have been testing since your last message.
Can you share an example request that is still failing? I can look into it on my end as well.
Hi @PjEdwards sorry for the wait, just needed to get the information together for you.
Did you fully depreciate "psm3-tmy" ? .
""Failed to find URL for resource_type = psm3-tmy""
Yes, the psm3-tmy data is quite out of date and had been listed in the deprecated section of the docs for several years already. The equivalent current dataset is https://developer.nrel.gov/docs/solar/nsrdb/nsrdb-GOES-tmy-v4-0-0-download/
Thank you for the help today. Things seem to be back up and running now. Will post if anything changes but thank you again.
Hello @PjEdwards: I believe this issue is occurring again today. Would you please be able to look into it and confirm? Thanks again!
Just getting back to my desk after some PTO. This is top of my list to fix...stay tuned.
OK, I have a clue. The NSRDB team went ahead with a plan to move some of the older datasets into an archival state. I was out of office so I didn't catch this in time to post advanced notice. The endpoints that no longer work (and I'm going to remove altogether this afternoon as well as update the docs) are:
https://developer.nrel.gov/docs/solar/nsrdb/psm3-2-2-download/ https://developer.nrel.gov/docs/solar/nsrdb/psm3-2-2-tmy-download/
@kjsauer Are these the endpoints you are having trouble with? If no can you share an example request that is failing?
Hello @PjEdwards: The URL that worked until late last week (approx. 5 days ago) was as follows: https://developer.nrel.gov/api/solar/nsrdb_psm3_download.csv I have experimented with a few other URLs today, but in every case I get the same message: "errors":["Data processing failure."]
P.S. For example, I also tried the following base URL and got the same error message: https://developer.nrel.gov/api/nsrdb/v2/solar/nsrdb-GOES-aggregated-v4-0-0-download.csv
The first example, using nsrdb_psm3_download.csv is one of the deprecated endpoints, so it is expected that this would fail. As of yesterday, when I deployed updates to the APIs to reflect the archived datasets, this endpoint now returns a 404 - Page Not Found error. I hope this will be more helpful for identifying deprecated vs. broken. The full list of deprecated endpoints (now that I've investigated thoroughly) is:
full-disc-download
psm3-2-2-tmy-download
psm-3-2-2-download
psm3-5min-download
psm3-tmy-download
psm3-download
psm-download
psm3_download
puerto-rico-download
ALL of these endpoints' location and temporal coverage is included (and expanded upon) by any of the GOES endpoints documented at https://developer.nrel.gov/docs/solar/nsrdb/ . You can explore their coverage areas in the web page at
https://nsrdb.nrel.gov/data-viewer
The second example, which is one of the new GOES datasets, is working fine for me this morning. Here is a sample that I've tested successfully (add your API Key). I tried different years, different attributes, different locations, etc. and haven't been able to produce a failure. Let me know if it is still broken for you, and if so could you provide me with a specific request that shows the error?
curl --location 'https://developer.nrel.gov/api/nsrdb/v2/solar/nsrdb-GOES-aggregated-v4-0-0-download.csv?api_key=.........&names=2024&wkt=POINT(-78%2035)&attributes=dni&email=paul.edwards%40nrel.gov'
Hello @PjEdwards: I tried the example URL you provided (thank you for providing that example!), and it works. I have not yet had a chance to update several code scripts, each with custom queries, so I may have follow-up questions, but if you want to close this ticket for now, that would be fine with me. By the way, I just tried the old URL again, and now it goes to a website that simply says "Not Found." So, it seems to have been updated in the last 24 hours. The documentation links also help to show that the old dataset is deprecated. Thanks again for all your support!