Posting incorrect date while ntp unavailable at boot

[rolfbartels]

I have found a bug where after a power outage the gateway is live before the internet connection is establised, this causes the gateway to capture data with the incorrect date as the ntp service is unable to update the system clock. the date it's using is the last updated date when the file system was writable "ipe-rw" and then posting to emonCMS with this timestamp. I also found that the for some reason the data initially posted was negative values for that date and a few results of negative data when the system started updating correctly.

A suggestion would be to either discard the data till the date has been updated or capture and offset until the date has been updated and then apply that offset to the first updated timestamp.

see attached log 2013-10-29 06:36:06,909 INFO Logging level set to DEBUG 2013-10-29 06:36:06,911 INFO Opening gateway... 2013-10-29 06:36:06,922 INFO Creating buffer emoncms_local 2013-10-29 06:36:06,933 INFO Creating buffer emoncms_remote 2013-10-29 06:36:06,937 INFO Creating listener RFM2Pi 2013-10-29 06:36:06,940 DEBUG Opening serial port: /dev/ttyAMA0 2013-10-29 06:36:06,951 INFO Setting send time interval to 300 2013-10-29 06:36:06,953 INFO Setting RFM2Pi | frequency: 4 2013-10-29 06:36:07,957 INFO Setting RFM2Pi | sgroup: 210 2013-10-29 06:36:08,960 INFO Setting RFM2Pi | baseid: 15 2013-10-29 06:36:09,964 INFO Creating listener Socket 2013-10-29 06:36:09,967 DEBUG Opening socket on port 50011 2013-10-29 06:36:09,989 DEBUG Broadcasting time: 6:36 2013-10-29 06:36:11,250 INFO Serial RX: 10 200 18 0 0 6 0 57 89 99 0 67 8 2013-10-29 06:36:11,253 DEBUG Node: 10 2013-10-29 06:36:11,255 DEBUG Values: [4808, 0, 6, 22841, 99, 2115] 2013-10-29 06:36:11,257 DEBUG Server emoncms.org -> buffer data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:36:11,260 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:36:11,262 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:11,264 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:11,266 INFO Sending to emoncms.org 2013-10-29 06:36:31,314 WARNING Couldn't send to server, URLError: [Errno -2] Name or service not known 2013-10-29 06:36:31,536 INFO Serial RX: > 0s 2013-10-29 06:36:31,538 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:36:31,541 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:31,543 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:31,545 INFO Sending to emoncms.org 2013-10-29 06:36:51,568 WARNING Couldn't send to server, URLError: [Errno -2] Name or service not known 2013-10-29 06:36:51,790 INFO Serial RX: -> 4 b 2013-10-29 06:36:51,793 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:36:51,795 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:51,797 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:36:51,800 INFO Sending to emoncms.org 2013-10-29 06:37:11,823 WARNING Couldn't send to server, URLError: [Errno -2] Name or service not known 2013-10-29 06:37:12,046 INFO Serial RX: 20 195 8 2013-10-29 06:37:12,048 DEBUG Node: 20 2013-10-29 06:37:12,050 DEBUG Values: [2243] 2013-10-29 06:37:12,053 DEBUG Server emoncms.org -> buffer data: [20, 2243], timestamp: 1383021432 2013-10-29 06:37:12,056 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:37:12,058 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:12,060 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:12,062 INFO Sending to emoncms.org 2013-10-29 06:37:32,085 WARNING Couldn't send to server, URLError: [Errno -2] Name or service not known 2013-10-29 06:37:32,313 INFO Serial RX: 10 175 18 0 0 0 0 66 89 100 0 43 8 2013-10-29 06:37:32,316 DEBUG Node: 10 2013-10-29 06:37:32,318 DEBUG Values: [4783, 0, 0, 22850, 100, 2091] 2013-10-29 06:37:32,320 DEBUG Server emoncms.org -> buffer data: [10, 4783, 0, 0, 22850, 100, 2091], timestamp: 1383021452 2013-10-29 06:37:32,323 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:37:32,325 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:32,327 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:32,330 INFO Sending to emoncms.org 2013-10-29 06:37:52,353 WARNING Couldn't send to server, URLError: [Errno -2] Name or service not known 2013-10-29 06:37:52,581 INFO Serial RX: 10 97 18 0 0 0 0 106 89 100 0 6 8 2013-10-29 06:37:52,583 DEBUG Node: 10 2013-10-29 06:37:52,586 DEBUG Values: [4705, 0, 0, 22890, 100, 2054] 2013-10-29 06:37:52,588 DEBUG Server emoncms.org -> buffer data: [10, 4705, 0, 0, 22890, 100, 2054], timestamp: 1383021472 2013-10-29 06:37:52,590 DEBUG Server emoncms.org -> send data: [10, 4808, 0, 6, 22841, 99, 2115], timestamp: 1383021371 2013-10-29 06:37:52,593 DEBUG Data string: &time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:52,595 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021371&node=10&json={1:4808,2:0,3:6,4:22841,5:99,6:2115} 2013-10-29 06:37:52,597 INFO Sending to emoncms.org 2013-10-29 06:37:53,764 DEBUG Send ok 2013-10-29 06:37:53,987 INFO Serial RX: 20 195 8 2013-10-29 06:37:53,989 DEBUG Node: 20 2013-10-29 06:37:53,991 DEBUG Values: [2243] 2013-10-29 06:37:53,994 DEBUG Server emoncms.org -> buffer data: [20, 2243], timestamp: 1383021473 2013-10-29 06:37:53,996 DEBUG Server emoncms.org -> send data: [20, 2243], timestamp: 1383021432 2013-10-29 06:37:53,998 DEBUG Data string: &time=1383021432&node=20&json={1:2243} 2013-10-29 06:37:54,000 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021432&node=20&json={1:2243} 2013-10-29 06:37:54,003 INFO Sending to emoncms.org 2013-10-29 06:37:55,137 DEBUG Send ok 2013-10-29 06:37:55,365 INFO Serial RX: 10 92 18 0 0 9 0 95 89 100 0 4 8 2013-10-29 06:37:55,367 DEBUG Node: 10 2013-10-29 06:37:55,370 DEBUG Values: [4700, 0, 9, 22879, 100, 2052] 2013-10-29 06:37:55,372 DEBUG Server emoncms.org -> buffer data: [10, 4700, 0, 9, 22879, 100, 2052], timestamp: 1383021475 2013-10-29 06:37:55,374 DEBUG Server emoncms.org -> send data: [10, 4783, 0, 0, 22850, 100, 2091], timestamp: 1383021452 2013-10-29 06:37:55,377 DEBUG Data string: &time=1383021452&node=10&json={1:4783,2:0,3:0,4:22850,5:100,6:2091} 2013-10-29 06:37:55,379 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021452&node=10&json={1:4783,2:0,3:0,4:22850,5:100,6:2091} 2013-10-29 06:37:55,381 INFO Sending to emoncms.org 2013-10-29 06:37:56,093 DEBUG Send ok 2013-10-29 06:37:56,304 INFO Serial RX: 10 74 18 0 0 4 0 92 89 100 0 253 7 2013-10-29 06:37:56,306 DEBUG Node: 10 2013-10-29 06:37:56,308 DEBUG Values: [4682, 0, 4, 22876, 100, 2045] 2013-10-29 06:37:56,311 DEBUG Server emoncms.org -> buffer data: [10, 4682, 0, 4, 22876, 100, 2045], timestamp: 1383021476 2013-10-29 06:37:56,313 DEBUG Server emoncms.org -> send data: [10, 4705, 0, 0, 22890, 100, 2054], timestamp: 1383021472 2013-10-29 06:37:56,316 DEBUG Data string: &time=1383021472&node=10&json={1:4705,2:0,3:0,4:22890,5:100,6:2054} 2013-10-29 06:37:56,318 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021472&node=10&json={1:4705,2:0,3:0,4:22890,5:100,6:2054} 2013-10-29 06:37:56,320 INFO Sending to emoncms.org 2013-10-29 06:37:58,129 DEBUG Send ok 2013-10-29 06:37:58,353 INFO Serial RX: 20 195 8 2013-10-29 06:37:58,355 DEBUG Node: 20 2013-10-29 06:37:58,357 DEBUG Values: [2243] 2013-10-29 06:37:58,360 DEBUG Server emoncms.org -> buffer data: [20, 2243], timestamp: 1383021478 2013-10-29 06:37:58,362 DEBUG Server emoncms.org -> send data: [20, 2243], timestamp: 1383021473 2013-10-29 06:37:58,364 DEBUG Data string: &time=1383021473&node=20&json={1:2243} 2013-10-29 06:37:58,366 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021473&node=20&json={1:2243} 2013-10-29 06:37:58,369 INFO Sending to emoncms.org 2013-10-29 06:37:59,185 DEBUG Send ok 2013-10-29 06:37:59,413 INFO Serial RX: 10 54 18 0 0 0 0 79 89 100 0 245 7 2013-10-29 06:37:59,416 DEBUG Node: 10 2013-10-29 06:37:59,418 DEBUG Values: [4662, 0, 0, 22863, 100, 2037] 2013-10-29 06:37:59,420 DEBUG Server emoncms.org -> buffer data: [10, 4662, 0, 0, 22863, 100, 2037], timestamp: 1383021479 2013-10-29 06:37:59,423 DEBUG Server emoncms.org -> send data: [10, 4700, 0, 9, 22879, 100, 2052], timestamp: 1383021475 2013-10-29 06:37:59,425 DEBUG Data string: &time=1383021475&node=10&json={1:4700,2:0,3:9,4:22879,5:100,6:2052} 2013-10-29 06:37:59,428 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021475&node=10&json={1:4700,2:0,3:9,4:22879,5:100,6:2052} 2013-10-29 06:37:59,430 INFO Sending to emoncms.org 2013-10-29 06:38:00,107 DEBUG Send ok 2013-10-29 06:38:00,312 INFO Serial RX: 20 195 8 2013-10-29 06:38:00,315 DEBUG Node: 20 2013-10-29 06:38:00,317 DEBUG Values: [2243] 2013-10-29 06:38:00,319 DEBUG Server emoncms.org -> buffer data: [20, 2243], timestamp: 1383021480 2013-10-29 06:38:00,322 DEBUG Server emoncms.org -> send data: [10, 4682, 0, 4, 22876, 100, 2045], timestamp: 1383021476 2013-10-29 06:38:00,324 DEBUG Data string: &time=1383021476&node=10&json={1:4682,2:0,3:4,4:22876,5:100,6:2045} 2013-10-29 06:38:00,327 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021476&node=10&json={1:4682,2:0,3:4,4:22876,5:100,6:2045} 2013-10-29 06:38:00,329 INFO Sending to emoncms.org 2013-10-29 06:38:00,988 DEBUG Send ok 2013-10-29 06:38:01,233 INFO Serial RX: 10 56 18 0 0 0 0 103 89 100 0 244 7 2013-10-29 06:38:01,235 DEBUG Node: 10 2013-10-29 06:38:01,238 DEBUG Values: [4664, 0, 0, 22887, 100, 2036] 2013-10-29 06:38:01,240 DEBUG Server emoncms.org -> buffer data: [10, 4664, 0, 0, 22887, 100, 2036], timestamp: 1383021481 2013-10-29 06:38:01,242 DEBUG Server emoncms.org -> send data: [20, 2243], timestamp: 1383021478 2013-10-29 06:38:01,245 DEBUG Data string: &time=1383021478&node=20&json={1:2243} 2013-10-29 06:38:01,247 DEBUG URL string: http://emoncms.org/input/post.json?apikey=xxxxxxxxxxxxxxxxxxxxxxxxxx6d66f&time=1383021478&node=20&json={1:2243} 2013-10-29 06:38:01,249 INFO Sending to emoncms.org 2013-10-29 06:38:02,258 DEBUG Send ok