Cloudiness Information

[nilsleh]

H @yuhao-nie and @ascott-20, in your paper table 5.1 you evaluate your models separately on cloudy and sunny days. However, this information is not natively included in your dataset to be downloaded. There is information in the preprocessing jupyter notebooks about cloudy and sunny days. However, when I do the following for the forecast task:

sunny_day = [(2017,9,15),(2017,10,6),(2017,10,22),(2018,2,16),(2018,6,12),(2018,6,23),(2019,1,25),(2019,6,23),(2019,7,14),(2019,10,14)]
cloudy_day = [(2017,6,24),(2017,9,20),(2017,10,11),(2018,1,25),(2018,3,9),(2018,10,4),(2019,5,27),(2019,6,28),(2019,8,10),(2019,10,19)]

sunny_datetime = [datetime.datetime(day[0],day[1],day[2]) for day in sunny_day]
cloudy_datetime = [datetime.datetime(day[0],day[1],day[2]) for day in cloudy_day]

arr = np.load("times_test_forecast.npy", allow_pickle=True)
date_arr = [val.date() for val in arr]
sunny_arr = [val.date() for val in sunny_datetime]
cloudy_arr = [val.date() for val in cloudy_datetime]

print(set(date_arr).intersection(set(sunny_arr)))
print(set(date_arr).intersection(set(cloudy_arr)))

The intersection with test forecasting dates and sunny dates is empty, suggesting there are no sunny test dates, only cloudy ones. However, you are reporting values for those in your paper.

For the nowcasting task, the above snippet yields, that the number of cloudy and sunny examples is about equal which is to be expected I guess. Could you help me out what I am missing?

Edit: The times_test_forecast.npy file is generated from running my script in #3