getWeatherForDate is limited to 398 rows

[semuelle]

Trying to gather daily weather data for a 2 year period, I call

getWeatherForDate("EDDW", station_type = "ID", start_date = "2013-01-01", end_date = "2014-12-31")

However, it never returns a data frame with more than 398 rows (sometimes 397), no matter what period I choose (I tried 2010 to 2015). Is this a limitation of the API?

[ghost]

Dear semuelle, a simple workaround is to lapply() the getWeatherForDate() over a range of years. using opt_write_to_file = TRUE works great. regards.

[Ram-N]

Hi Semulle, Laurent's suggestion would work. However, I think there is something bigger going on here.

1. THe package doesn't use any of WU's native API's.
2. I wonder if this is somehow because you are fetching for an "ID" as opposed to an airport. Do you mind trying this for an airport and see if the 398 limit goes away? If so, there is a bug and I have to find time to work on it.

Thanks so much for bringing this to my attention. Ram

[semuelle]

Hi Ram and Laurent, this was the issue of a student, and it was easily worked around. I just thought I'd bring it to your attention because it didn't seem like intended behavior. The issue persists with airport codes:

getWeatherForDate("KHWO", station_type = "airportCode", start_date = "2013-01-01", end_date = "2014-12-31")
Checking Summarized Data Availability For KHWO
Found 398 records for 2013-01-01 to 2014-12-31

[ghost]

Having spent a few hours trying to debug my code and then getSummarizedWeather() and the functions it calls before finding this, I'm in agreement that it would be nice to fix this. If that's hard (and it might be, for it might require using the APIs with a key), then it might be nice either to test the date range in weatherData and report an error or a warning when more than 398 rows are being requested or possibly to design Laurent's solution into weatherData.

Thanks for weatherData, BTW.

[Maiae]

Hi there, Thanks for this package. It is really useful. Is there any updates on the number of records being pulled when using getSummarizedWeather ?

[Ram-N]

All:

This limitation is coming from WU. They don't want us pulling large volumes of data in one go. As others have suggested, a loop (or lapply) to get data in chunks works.

To address this, I will do the following:

1. If the number of days in the date range is greater than 365, I will try and internally make multiple calls. 1a. I will also warn the users as appropriate.
2. I will also add a new wrapper function to fetch the dataset for very large (multi-year) time spans.

With these two changes, I can then close this issue. (For now, I am letting it be open until I address these.)