pecan
pecan copied to clipboard
PecanProject
Possible bug: PEcAn output sometime repeats first/last date/time between years or starts at -1
Describe the bug When working on running some test runs of ED2 through ILAMB we discovered that, at least in my output, the last date of the year in time variable is the first date of the next years output
If we look at the times of my output from 2001-2002 for example
0 55115.0104173 55480.0104173 17520
1 55480.0104173 55845.0104173 17520
you can see the time is the same here as well.....this could cause us problems for processing that attempts to chain output together, such as with ILAMB currently
Also, while playing with SIPNET output I realized that in my runs the first value of the time variable is -1!
data:
time = -1, -0.958333333333333, -0.916666666666667, -0.875,
-0.833333333333333, -0.791666666666667, -0.75, -0.708333333333333,
-0.666666666666667, -0.625, -0.583333333333333, -0.541666666666667, -0.5,
-0.458333333333333, -0.416666666666667, -0.375, -0.333333333333333,
-0.291666666666667, -0.25, -0.208333333333333,
time = UNLIMITED ; // (8760 currently)
...
double time(time) ;
time:units = "days since 2000-01-01 00:00:00" ;
time:long_name = "time" ;
time:calendar = "standard" ;
Expected behavior
I would think that we do not want the output provided in this way and instead only have the time variable include that years data/date range? For example, ending in fractional time such as: 364.895833333333, 364.916666666667, 364.9375, 364.958333333333, 364.979166666667 ;
And starting with 0.04166667 for example if hourly (i.e. SIPNET).
Need to check if this is coming from met or from model2netcdf....I suspect m2n step
Machine (please complete the following information):
- Server modex
- OS: linux
- Browser(if applicable) NA
- Version NA
Here is an example of raw SIPNET output
Notes: (PlantWoodC, PlantLeafC, Soil and Litter in g C/m^2; Water and Snow in cm; SoilWetness is fraction of WHC;
loc year day time plantWoodC plantLeafC soil microbeC coarseRootC fineRootC litter litterWater soilWater soilWetnessFrac snow npp nee cumNEE gpp rAboveground rSoil rRoot ra rh rtot evapotranspiration fluxestranspiration fPAR
0 2000 0 0.00 17998.95 0.00 10001.44 5.00 5999.96 5999.59 280.00 0.500 6.00 0.500 1.00 -0.02 0.06 0.06 0.00 0.019 0.041 0.005 0.024 0.036 0.059 0.00323719 0.0000 0.0000
0 2000 0 1.00 17997.91 0.00 10002.85 5.00 5999.91 5999.17 280.00 0.500 6.00 0.500 0.99 -0.02 0.09 0.15 0.00 0.018 0.072 0.005 0.023 0.068 0.090 0.00283669 0.0000 0.0000
0 2000 0 2.00 17996.86 0.00 10004.27 5.00 5999.87 5998.76 280.00 0.500 6.00 0.500 0.99 -0.02 0.09 0.24 0.00 0.017 0.070 0.005 0.022 0.066 0.087 0.00275556 0.0000 0.0000
0 2000 0 3.00 17995.82 0.00 10005.71
and the end of that year
0 2000 364 23.00 10911.18 178.24 18025.82 5.00 5591.78 3376.22 280.00 0.500 6.47 0.539 0.04 -0.02 0.10 1951.40 0.00 0.014 0.084 0.005 0.019 0.079 0.097 0.00636472 0.0000 0.3524
0 2001 0 0.00 10910.54 178.23 18026.64 5.00 5591.74 3375.98 280.00 0.500 6.47 0.539 0.03 -0.02 0.10 1951.50 0.00 0.012 0.084 0.005 0.017 0.078 0.095 0.00632517 0.0000 0.3524
0 2001 0 1.00 10909.91 178.23 18027.46 5.00 5591.70 3375.75 280.00 0.500 6.47 0.539 0.02 -0.02 0.10 1951.60 0.00 0.011 0.084 0.005 0.016 0.079 0.095 0.00691700 0.0000 0.3524
0 2001 0 2.00 10909.28 178.22 18028.28 5.00 5591.66 3375.52 280.00 0.500
so that looks ok
WRT SIPNET the issue is here
t <- ncdf4::ncdim_def(name = "time",
units = paste0("days since ", y, "-01-01 00:00:00"),
vals = sub.sipnet.output$day - 1 + (sub.sipnet.output$time/24),
calendar = "standard",
unlim = TRUE)
and I am not certain why we have the -1 there
> vals = sub.sipnet.output$day - 1 + (sub.sipnet.output$time/24)
> head(vals)
[1] -1.0000000 -0.9583333 -0.9166667 -0.8750000 -0.8333333 -0.7916667
versus
> vals = sub.sipnet.output$day + (sub.sipnet.output$time/24)
> head(vals)
[1] 0.00000000 0.04166667 0.08333333 0.12500000 0.16666667 0.20833333
where the second would end as
> tail(vals)
[1] 363.7500 363.7917 363.8333 363.8750 363.9167 363.9583
With ED2 its a bit tricky because in python (ILAMB):
from netCDF4 import num2date,date2num
print num2date(365.,"days since 2001-01-01 00:00:00")
print num2date( 0.,"days since 2002-01-01 00:00:00")
gives
2002-01-01 00:00:00
2002-01-01 00:00:00
and we currently provide ED2 output time such as:
time = 0, 0.0208345225184086, 0.0416690450368172, 0.0625035675552258,
0.0833380900736343, 0.104172612592043, 0.125007135110452,
0.14584165762886, 0.166676180147269, 0.187510702665677,
0.208345225184086, 0.229179747702494, 0.250014270220903,
...
364.333295279411, 364.354129801929, 364.374964324448, 364.395798846966,
364.416633369485, 364.437467892003, 364.458302414521, 364.47913693704,
364.499971459558, 364.520805982077, 364.541640504595, 364.562475027113,
364.583309549632, 364.60414407215, 364.624978594669, 364.645813117187,
364.666647639705, 364.687482162224, 364.708316684742, 364.729151207261,
364.749985729779, 364.770820252297, 364.791654774816, 364.812489297334,
364.833323819853, 364.854158342371, 364.87499286489, 364.895827387408,
364.916661909926, 364.937496432445, 364.958330954963, 364.979165477482,
365 ;
so it seems we need to be ending in the 364. part for non leap and 365. for leap, for example
>>> from netCDF4 import num2date,date2num
>>> print num2date(365.,"days since 2004-01-01 00:00:00")
2004-12-31 00:00:00
for a leap year where
>>> print num2date(366.,"days since 2004-01-01 00:00:00")
2005-01-01 00:00:00
results in the same issue I raised above
For ED2 I propose we change to:
> days <- seq(0,364,0.02083333)
> head(days)
[1] 0.00000000 0.02083333 0.04166666 0.06249999 0.08333332 0.10416665
> tail(days)
[1] 363.8958 363.9166 363.9374 363.9583 363.9791 363.9999
and
> days <- seq(0,365,0.02083333)
> head(days)
[1] 0.00000000 0.02083333 0.04166666 0.06249999 0.08333332 0.10416665
> tail(days)
[1] 364.8958 364.9166 364.9374 364.9583 364.9791 364.9999
which would still translate to correct cal dates, eg
> head(as.Date(days, origin=as.Date("2001-01-01")))
[1] "2001-01-01" "2001-01-01" "2001-01-01" "2001-01-01" "2001-01-01" "2001-01-01"
> tail(as.Date(days, origin=as.Date("2001-01-01")))
[1] "2001-12-31" "2001-12-31" "2001-12-31" "2001-12-31" "2001-12-31" "2001-12-31"
for leap years. @istfer @tonygardella @ashiklom thoughts? Obviously we would set the timestep dynamically based on output frequency instead of hard-coding but I think we just need to change the end date from 365 to 364 or 365 from 366 depending.
@serbinsh I haven't seen the SIPNET issue before. I just checked a SIPNET output and it starts from day 1, could it be a version thing?
> head(sipnet.output)
loc year day time plantWoodC plantLeafC soil microbeC coarseRootC fineRootC
1 0 2005 1 0.0 8819.99 0 1600.18 0.8 2519.93 1259.91
2 0 2005 1 0.5 8819.97 0 1600.36 0.8 2519.85 1259.81
3 0 2005 1 1.0 8819.96 0 1600.54 0.8 2519.78 1259.72
therefore:
> vals = sub.sipnet.output$day - 1 + (sub.sipnet.output$time/24)
> head(vals)
[1] 0.00000000 0.02083333 0.04166667 0.06250000 0.08333333 0.10416667
> tail(vals)
[1] 364.8750 364.8958 364.9167 364.9375 364.9583 364.9792
> t <- ncvar_get(nc, "time")
> head(t)
[1] 0.00000000 0.02083333 0.04166667 0.06250000 0.08333333 0.10416667
> tail(t)
[1] 364.8750 364.8958 364.9167 364.9375 364.9583 364.9792
or could it be something related to the the met2model? I think SIPNET takes the time in the sipnet.clim file into consideration. The clim files I'm using are generated a while ago, maybe latest changes affected these
@istfer what version of sipnet are you using that gives you day as 1? I was running r136 or whatever NOT "unk" but yeah that looks like a version issue! and that could cause us issues unless we determine 0 vs 1
@istfer OK so narrowing down, here is my input sipnet.clim file
0 2000 0 0 0.04166667 -0.433 0.376 0.000 0.000 100.433 136.257 491.841 3.572 0.291
0 2000 0 1 0.04166667 -1.022 -0.128 0.000 0.000 68.020 106.262 499.302 3.362 0.291
0 2000 0 2 0.04166667 -1.606 -0.421 0.000 0.000 43.585 92.809 499.950 3.287 0.291
0 2000 0 3 0.04166667 -2.558 0.153 0.000 0.000 19.511 130.896 487.128 2.770 0.291
0 2000 0 4 0.04166667 -3.706 0.204 0.000 0.000 4.198 159.356 460.947 2.457 0.291
0 2000 0 5 0.04166667 -4.769 0.110 0.000 0.000 5.600 192.289 423.814 2.641 0.291
0 2000 0 6 0.04166667 -5.526 -0.282 0.000 0.000 8.120 201.453 397.363 2.653 0.291
0 2000 0 7 0.04166667 -5.961 -0.044 0.000 0.000 5.246 222.228 387.034 2.306 0.291
0 2000 0 8 0.04166667 -5.873 -0.396 0.199 0.000 4.868 203.824 390.057 2.721 0.291
0 2000 0 9 0.04166667 -5.133 0.130 0.401 0.000 11.418 210.657 406.345 2.904 0.291
0 2000 0 10 0.04166667 -4.606 -0.012 0.574 0.000 26.797 202.741 407.945 2.638 0.291
0 2000 0 11 0.04166667 -4.362 0.211 0.638 0.000 39.046 216.858 403.749 2.025 0.291
0 2000 0 12 0.04166667 -4.048 -0.200 0.739 0.000 49.391 198.420 403.986 2.063 0.291
0 2000 0 13 0.04166667 -3.649 -0.253 0.607 0.000 55.847 188.786 411.273 1.289 0.291
0 2000 0 14 0.04166667 -3.626 0.131 0.437 0.000 59.100 208.189 408.843 1.734 0.291
0 2000 0 15 0.04166667 -3.930 -0.080 0.157 0.000 60.933 211.205 396.469 1.616 0.291
0 2000 0 16 0.04166667 -4.145 -0.644 0.012 0.000 62.895 196.041 387.170 1.496 0.291
0 2000 0 17 0.04166667 -4.443 0.105 0.000 0.000 61.198 236.942 378.919 1.808 0.291
0 2000 0 18 0.04166667 -4.518 0.050 0.000 0.000 60.600 236.412 377.020 2.160 0.291
0 2000 0 19 0.04166667 -4.623 -0.135 0.000 0.000 51.999 223.087 382.171 2.078 0.291
0 2000 0 20 0.04166667 -4.592 -0.074 0.000 0.000 51.546 224.259 383.645 2.273 0.291
0 2000 0 21 0.04166667 -4.431 -0.966 0.000 0.000 48.845 177.983 391.666 2.294 0.291
0 2000 0 22 0.04166667 -4.274 0.152 0.000 0.000 42.899 215.100 402.855 2.563 0.291
0 2000 0 23 0.04166667 -4.235 -0.533 0.000 0.000 30.999 171.920 416.050 2.832 0.291
0 2000 1 0 0.04166667 -4.553 -0.300 0.000 0.000 5.512 167.050 430.972 2.746 0.291
0 2000 1 1 0.04166667 -5.150 0.217 0.000 0.000 0.000 203.668 417.228 2.489 0.291
0 2000 1 2 0.04166667 -5.506 0.100 0.000 0.621 0.000 209.524 406.124 2.560 0.291
0 2000 1 3 0.04166667 -5.688 0.146 0.000 1.863 0.000 217.156 400.534 2.391 0.291
0 2000 1 4 0.04166667 -5.837 0.080 0.000 1.087 0.000 218.738 396.020 2.705 0.291
0 2000 1 5 0.04166667 -5.902 -0.194 0.000 0.932 0.000 208.559 394.068 2.522 0.291
0 2000 1 6 0.04166667 -5.878 -0.203 0.000 0.621 0.000 207.457 394.789 2.543 0.291
0 2000 1 7 0.04166667 -5.763 0.370 0.000 2.174 0.000 229.550 398.248 2.425 0.291
0 2000 1 8 0.04166667 -5.337 -0.092 0.122 2.018 0.000 195.760 411.378 2.577 0.291
0 2000 1 9 0.04166667 -4.868 -0.526 0.259 0.932 0.000 162.046 426.224 3.651 0.291
0 2000 1 10 0.04166667 -4.800 0.038 0.332 0.466 0.000 184.481 428.415 3.601 0.291
0 2000 1 11 0.04166667 -4.206 0.666 0.402 0.621 0.000 193.298 448.059 3.630 0.291
0 2000 1 12 0.04166667 -3.661 -0.022 0.839 0.776 0.000 143.512 466.706 4.431 0.291
0 2000 1 13 0.04166667 -3.334 -0.717 1.134 0.466 12.835 114.715 465.410 4.670 0.291
0 2000 1 14 0.04166667 -3.183 -0.172 0.858 0.311 42.649 162.607 441.002 4.792 0.291
0 2000 1 15 0.04166667 -3.339 -0.469 0.267 0.311 39.049 151.700 439.005 4.519 0.291
0 2000 1 16 0.04166667 -3.630 0.845 0.024 0.000 24.439 206.330 443.367 4.987 0.291
0 2000 1 17 0.04166667 -4.063 0.541 0.000 0.000 28.848 211.587 424.012 4.044 0.291
0 2000 1 18 0.04166667 -4.466 0.473 0.000 0.000 27.139 220.310 412.203 3.689 0.291
0 2000 1 19 0.04166667 -4.892 0.257 0.000 0.000 28.478 225.690 396.983 3.625 0.291
0 2000 1 20 0.04166667 -5.524 -0.287 0.000 0.000 30.128 223.151 375.420 2.896 0.291
0 2000 1 21 0.04166667 -6.071 -0.317 0.000 0.000 30.071 238.344 358.946 2.453 0.291
0 2000 1 22 0.04166667 -6.380 -0.827 0.000 0.000 26.396 221.945 353.545 2.264 0.291
0 2000 1 23 0.04166667 -6.704 -0.480 0.000 0.000 31.298 250.912 339.312 2.532 0.291
so you can see starts with 0
and
0 2001 363 0 0.04166667 -13.042 1.509 0.000 0.000 46.950 503.577 177.913 3.800 0.312
0 2001 363 1 0.04166667 -12.941 1.504 0.000 0.000 45.600 500.112 181.100 4.200 0.312
0 2001 363 2 0.04166667 -12.740 1.513 0.000 0.000 44.600 495.852 185.813 4.400 0.312
0 2001 363 3 0.04166667 -12.539 1.504 0.000 0.000 45.400 492.436 188.783 3.900 0.312
0 2001 363 4 0.04166667 -12.439 1.504 0.000 0.000 47.400 492.533 188.686 4.100 0.312
0 2001 363 5 0.04166667 -12.338 1.512 0.000 0.000 57.550 501.142 180.455 4.700 0.312
0 2001 363 6 0.04166667 -12.539 1.499 0.000 0.000 63.100 509.912 171.083 4.800 0.312
0 2001 363 7 0.04166667 -12.740 1.479 0.000 0.000 52.500 502.086 177.913 4.200 0.312
0 2001 363 8 0.04166667 -13.042 1.499 0.206 0.000 59.450 515.589 165.413 4.200 0.312
0 2001 363 9 0.04166667 -13.243 1.494 0.691 0.000 62.000 521.530 159.228 4.800 0.312
0 2001 363 10 0.04166667 -13.343 1.484 1.532 0.000 66.550 527.344 152.882 5.600 0.312
0 2001 363 11 0.04166667 -12.841 1.491 2.401 0.000 79.700 531.757 148.849 5.600 0.312
0 2001 363 12 0.04166667 -12.037 1.490 2.606 0.000 96.200 532.887 147.644 5.200 0.312
0 2001 363 13 0.04166667 -11.132 1.474 2.302 0.311 119.050 536.689 143.088 4.800 0.312
0 2001 363 14 0.04166667 -10.530 1.490 1.501 0.466 137.650 543.210 137.355 5.200 0.312
0 2001 363 15 0.04166667 -10.530 1.484 0.598 0.000 139.250 544.520 135.755 4.800 0.312
0 2001 363 16 0.04166667 -11.434 1.475 0.044 0.000 115.600 539.515 140.309 4.200 0.312
0 2001 363 17 0.04166667 -12.539 1.489 0.000 0.000 95.500 541.824 138.683 3.900 0.312
0 2001 363 18 0.04166667 -13.443 1.489 0.000 0.000 87.500 550.330 130.147 4.000 0.312
0 2001 363 19 0.04166667 -14.147 1.477 0.000 0.000 79.300 553.688 126.216 4.100 0.312
0 2001 363 20 0.04166667 -14.649 1.472 0.000 0.000 80.200 562.654 117.024 4.400 0.312
0 2001 363 21 0.04166667 -15.252 1.475 0.000 0.000 74.850 566.990 112.819 4.500 0.312
0 2001 363 22 0.04166667 -15.754 1.482 0.000 0.000 69.950 570.086 110.076 4.300 0.312
0 2001 363 23 0.04166667 -16.056 1.468 0.000 0.000 68.400 572.311 107.174 4.000 0.312
0 2001 364 0 0.04166667 -16.458 1.473 0.000 0.000 67.050 576.955 102.742 4.600 0.312
0 2001 364 1 0.04166667 -17.060 1.465 0.000 0.000 62.900 580.796 98.538 4.300 0.312
0 2001 364 2 0.04166667 -17.462 1.471 0.000 0.000 62.000 585.566 94.075 3.800 0.312
0 2001 364 3 0.04166667 -17.965 1.469 0.000 0.000 59.900 589.826 89.696 3.100 0.312
0 2001 364 4 0.04166667 -18.568 1.462 0.000 0.000 58.400 595.426 83.740 2.800 0.312
0 2001 364 5 0.04166667 -18.969 1.466 0.000 0.000 56.500 598.520 80.855 2.500 0.312
0 2001 364 6 0.04166667 -19.472 1.466 0.000 0.000 53.500 601.281 78.078 3.100 0.312
0 2001 364 7 0.04166667 -19.170 1.455 0.000 0.000 52.200 596.011 82.817 2.100 0.312
0 2001 364 8 0.04166667 -18.869 1.457 0.320 0.000 51.050 591.428 87.487 2.900 0.312
0 2001 364 9 0.04166667 -17.563 1.451 1.287 0.000 62.000 585.859 92.759 2.800 0.312
0 2001 364 10 0.04166667 -15.352 1.449 2.086 0.000 72.900 565.308 113.218 2.800 0.312
0 2001 364 11 0.04166667 -13.443 1.452 2.564 0.000 88.950 550.002 128.697 2.800 0.312
0 2001 364 12 0.04166667 -11.836 1.444 2.678 0.000 114.950 545.463 132.859 2.100 0.312
0 2001 364 13 0.04166667 -10.831 1.440 2.293 0.000 130.700 540.313 137.802 3.000 0.312
0 2001 364 14 0.04166667 -11.333 1.433 1.492 0.155 121.000 540.789 136.971 3.200 0.312
0 2001 364 15 0.04166667 -12.740 1.432 0.512 0.000 92.950 540.250 137.463 3.600 0.312
0 2001 364 16 0.04166667 -13.443 1.424 0.029 0.000 80.850 540.538 136.797 3.200 0.312
0 2001 364 17 0.04166667 -13.544 1.425 0.000 0.000 77.200 538.698 138.675 2.700 0.312
0 2001 364 18 0.04166667 -12.303 1.425 0.000 0.000 52.600 491.271 186.088 3.166 0.312
0 2001 364 19 0.04166667 -12.374 1.421 0.000 0.000 44.149 484.044 193.164 2.871 0.312
0 2001 364 20 0.04166667 -13.032 1.416 0.000 0.000 39.277 491.198 185.761 2.769 0.312
0 2001 364 21 0.04166667 -14.262 1.412 0.000 0.000 34.911 508.061 168.674 3.102 0.312
0 2001 364 22 0.04166667 -14.844 1.404 0.000 0.000 30.150 512.417 163.932 3.406 0.312
0 2001 364 23 0.04166667 -15.402 1.403 0.000 0.000 27.243 518.203 158.108 2.669 0.312
and ends at 364. Does yours end with 365 by chance for a non leap?
Strangely it looks like my runs did not have leap met
0 2004 363 23 0.04166667 -8.001 1.312 0.000 0.000 2.650 339.257 332.666 1.706 0.308
0 2004 364 0 0.04166667 -8.102 1.282 0.000 0.000 2.798 340.570 329.895 1.882 0.308
0 2004 364 1 0.04166667 -8.179 0.948 0.000 0.000 2.600 326.437 328.132 1.858 0.308
0 2004 364 2 0.04166667 -8.185 1.356 0.000 0.000 3.499 346.979 327.062 1.994 0.308
0 2004 364 3 0.04166667 -8.279 1.454 0.000 0.000 3.650 354.288 324.500 2.441 0.308
0 2004 364 4 0.04166667 -8.084 1.436 0.000 0.000 3.144 347.881 330.015 2.747 0.308
0 2004 364 5 0.04166667 -7.790 1.409 0.000 0.000 2.794 338.550 338.059 2.575 0.308
0 2004 364 6 0.04166667 -7.425 1.382 0.000 0.000 1.942 326.654 348.660 2.749 0.308
0 2004 364 7 0.04166667 -6.757 1.374 0.000 0.000 0.116 305.884 368.999 3.290 0.308
0 2004 364 8 0.04166667 -5.497 1.369 0.052 0.000 0.000 267.967 406.701 3.472 0.308
0 2004 364 9 0.04166667 -3.161 1.353 0.067 0.000 0.000 188.884 484.994 4.810 0.308
0 2004 364 10 0.04166667 -1.557 1.341 0.145 0.466 0.000 127.691 545.617 5.119 0.308
0 2004 364 11 0.04166667 -0.740 1.330 0.138 1.087 0.000 93.617 579.172 2.841 0.308
0 2004 364 12 0.04166667 -0.388 1.283 0.155 1.863 0.000 76.253 594.238 2.865 0.308
0 2004 364 13 0.04166667 0.049 0.950 0.138 1.242 0.000 41.232 613.402 6.940 0.308
0 2004 364 14 0.04166667 0.994 1.338 0.119 2.951 0.000 16.352 656.804 6.072 0.308
0 2004 364 15 0.04166667 2.087 1.468 0.053 0.932 0.000 -30.885 710.365 5.627 0.308
0 2004 364 16 0.04166667 2.681 1.463 0.001 0.621 0.000 -61.660 740.909 4.882 0.308
0 2004 364 17 0.04166667 2.867 1.445 0.000 0.776 0.000 -72.437 750.800 4.829 0.308
0 2004 364 18 0.04166667 3.600 1.435 0.000 0.776 0.000 -112.841 790.720 2.381 0.308
0 2004 364 19 0.04166667 4.299 1.430 0.000 0.466 0.000 -152.990 830.614 2.522 0.308
0 2004 364 20 0.04166667 5.669 1.433 0.000 0.466 0.000 -235.950 913.729 2.470 0.309
0 2004 364 21 0.04166667 6.689 1.424 0.000 0.310 5.008 -298.007 975.312 6.086 0.309
0 2004 364 22 0.04166667 6.645 1.409 0.000 0.155 13.670 -287.069 963.670 4.143 0.310
0 2004 364 23 0.04166667 6.032 1.435 0.000 0.155 53.540 -205.478 883.343 4.816 0.311
yes, for a non-leap year my clim starts with 1 and ends with 365
> head(clim)
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14
1 0 2005 1 0.0 0.02083333 9.981012 -0.1449951 0 0 336 0 895.1882 1 0.6
2 0 2005 1 0.5 0.02083333 10.042993 -0.1409973 0 0 347 0 889.3658 1 0.6
> tail(clim)
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14
122683 0 2011 365 21.0 0.02083333 5.499994 0.4399963 0 0 89 0 815.5093 1 0.6
122684 0 2011 365 21.5 0.02083333 5.480005 0.4500061 0 0 98 0 805.2459 1 0.6
122685 0 2011 365 22.0 0.02083333 5.260004 0.4599854 0 0 83 0 806.4457 1 0.6
122686 0 2011 365 22.5 0.02083333 5.059991 0.4699951 0 0 69 0 808.0639 1 0.6
122687 0 2011 365 23.0 0.02083333 5.219995 0.4699951 0 0 79 0 807.9565 1 0.6
122688 0 2011 365 23.5 0.02083333 5.390009 0.4500061 0 0 92 0 805.5776 1 0.6
@istfer what if we implement something like:
if (tail(unique(sipnet.output$day), n=1)==365) {
tvals <- sub.sipnet.output$day-1 + (sub.sipnet.output$time/out.day)
} else {
tvals <- sub.sipnet.output$day + (sub.sipnet.output$time/out.day)
}
t <- ncdf4::ncdim_def(name = "time",
units = paste0("days since ", y, "-01-01 00:00:00"),
vals = tvals,
calendar = "standard",
unlim = TRUE)
point is to be consistent with 0-364/365 depending on non/leap instead of some being 1-365/366
but then how to deal with runs less than a year....
That also won't work for shorter runs. Don't forget that we've now got 16 day SIPNET forecasts running daily at Willow Creek
is this harder than it should be because we are inconsistent with how met days are specified to SIPNET? That is sometimes met has a 0 start and sometimes its indexed based on a start day of 1? Can we just pick one? That said, not sure how best to make this back compatible....but as it is I dont think we want to have both. I suppose we could look at the met file but then again it may not have start/end year dates if just a few weeks of met drivers. I dont know the solution right now....
If the issue is the difference between runk and r136 (and moving forward) then I'm fine with breaking backward compatibility with runk, dropping it from the VM, and labeling it as deprecated. There's really no difference in model guts between the two and r136 has some I/O bug fixes.
Day of year should start with 1 to represent Jan 1. Also watch out for timezone shifts. Shifting from UTC to CST or EST or vice versa is another source of error in terms of when start/end days don't get parsed properly in input drivers or outputs. Ameriflux files are in local time, so things get shifted forward during conversions. This leads to days getting added or removed.
@mdietze I dont think its a version issue, at least for SIPNET, it looks more like an issue of how the met drivers are specified.
@ankurdesai issue is that 0 and 365 get parsed as the same date: https://github.com/PecanProject/pecan/issues/2094#issuecomment-418801635
What we need is a standard way we define time in netCDF, which includes days since, calendar type, and how the time values are specified such that we avoid the 0/365/366 issue as being read as the same date and thus duplicated. Obviously when we use leap we need to specify a leap cal and if not probably should specify a no_leap cal but also we should be consistent that values should be from 1.x to 364.x (or 365.x if leap) for all model output
this is again another rabbit hole i didnt anticipate when starting this discussion...I thought perhaps it would be a fairly quick fix but it seems it actually is not
It would seem that we actually want to first determine the posix date and then generate the DOY tvals from this.... and also that we need to start with 0, e.g.
> as.Date(300, origin="2001-01-01")
[1] "2001-10-28"
> as.Date(364, origin="2001-01-01")
[1] "2001-12-31"
> as.Date(1, origin="2001-01-01")
[1] "2001-01-02"
> as.Date(0, origin="2001-01-01")
[1] "2001-01-01"
> as.Date(365, origin="2001-01-01")
[1] "2002-01-01"
...or perhaps we start to use time bounds, e.g.
double time(time) ;
time:units = "days since 1850-01-01 00:00:00" ;
time:calendar = "noleap" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
time:bounds = "time_bounds" ;
double time_bounds(time, nv) ;
time_bounds:long_name = "time_bounds" ;
time_bounds:standard_name = "time_bounds" ;
Sounds like we've concluded that years start at 0. Are there other things to resolve here? (other than some code fixes and tests to make sure ever met2CF, met2model, and model2netcdf are doing this correctly)
@mdietze take a look at https://github.com/PecanProject/pecan/pull/2095 Probably still some cleanup but I have tested on modex (command and via web) and it doesnt break PEcAn but does provide the additional "time_bounds" that is common in global ESM output and thus something programs like ILAMB looks for. Provides to start/end timepoints for each timestep which means we use time or time_bounds to define the comparison with data, etc. Where time_bounds can be used aggregate data of higher frequency into exact model output bins..etc. And as Nate suggested our outputs arent instantaneous and as such bounds provide explicit bounds for each step.
either way I was hoping that we could add to the output since it doesnt impact existing functions but could add functionality in the future and provides an easier connection with ILAMB instead of have to come up with a workaround to read "time" and make assumptions in the comparison with benchmarks.....happy to explain more tomorrow
For example, global CLMCN output
netcdf rsds_Amon_CLM40cn_historical_r1i1p1_185001-201012 {
dimensions:
time = UNLIMITED ; // (1932 currently)
lat = 192 ;
lon = 288 ;
levgrnd = 15 ;
levlak = 10 ;
hist_interval = 2 ;
variables:
float lat(lat) ;
lat:long_name = "coordinate latitude" ;
lat:units = "degrees_north" ;
lat:_FillValue = 1.e+36f ;
lat:missing_value = 1.e+36f ;
float levgrnd(levgrnd) ;
levgrnd:long_name = "coordinate soil levels" ;
levgrnd:units = "m" ;
float levlak(levlak) ;
levlak:long_name = "coordinate lake levels" ;
levlak:units = "m" ;
float lon(lon) ;
lon:long_name = "coordinate longitude" ;
lon:units = "degrees_east" ;
lon:_FillValue = 1.e+36f ;
lon:missing_value = 1.e+36f ;
float time(time) ;
time:long_name = "time" ;
time:units = "days since 1850-01-01 00:00:00" ;
time:calendar = "noleap" ;
time:bounds = "time_bounds" ;
double time_bounds(time, hist_interval) ;
time_bounds:long_name = "history time interval endpoints" ;
One thing I would prefer is if we could remove int variable from the output and just have the dimensional variable.
this is how SIPNET is written in my PR
[sserbin@modex 2000078549]$ ncdump -h 2001.nc
netcdf \2001 {
dimensions:
time_interval = 2 ;
time = UNLIMITED ; // (8760 currently)
lon = 1 ;
lat = 1 ;
variables:
int time_interval(time_interval) ;
time_interval:long_name = "output time interval endpoints" ;
double time(time) ;
time:units = "days since 2001-01-01 00:00:00" ;
time:long_name = "time" ;
time:calendar = "standard" ;
time:bounds = "time_bounds" ;
double time_bounds(time, time_interval) ;
time_bounds:units = "days since 2001-01-01 00:00:00" ;
time_bounds:long_name = "output time bounds" ;
and here is CLM-FATES output
float time(time) ;
time:long_name = "time" ;
time:units = "days since 1900-01-01 00:00:00" ;
time:calendar = "noleap" ;
time:bounds = "time_bounds" ;
int mcdate(time) ;
mcdate:long_name = "current date (YYYYMMDD)" ;
int mcsec(time) ;
mcsec:long_name = "current seconds of current date" ;
mcsec:units = "s" ;
int mdcur(time) ;
mdcur:long_name = "current day (from base day)" ;
int mscur(time) ;
mscur:long_name = "current seconds of current day" ;
int nstep(time) ;
nstep:long_name = "time step" ;
double time_bounds(time, hist_interval) ;
time_bounds:long_name = "history time interval endpoints" ;
So I guess the questions are 1) are we happy with inclusion of a bounds variable, 2) if so, what shall we call it (seems for CLM its common to call it "hist_interval" of size 2) with a longname of "history time interval endpoints" and no units; 3) if we keep that variable how we avoid getting the
int time_interval(time_interval) ;
time_interval:long_name = "output time interval endpoints" ;
showing up? ....and whatever else I might be missing/forgetting.
This issue is stale because it has been open 365 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}