vzlogger icon indicating copy to clipboard operation
vzlogger copied to clipboard

Published 20 hours ago • verified publisher icon volkszaehler

Vzlogger only reads first 32 lines of data file

Open drtorchwood opened this issue 2 years ago • 2 comments

Hi, I use an external python script to get weather data from the DWD. I download the data once per day with 10 min resolution. Now, I want to read the file with vzlogger. In principle, the configuration is working but it only reads the first 32 lines.

This is the part of the config:

    {
      "enabled": true,
      "allowskip": true,
      "interval": 600,
      "aggtime": -1,
      "aggfixedinterval": false,
      "channels": [
        {
          "api": "volkszaehler",
          "uuid": "f7fa6f00-cee8-11ec-ba4a-579e870e7bf9",
          "identifier": "radiation_global",
          "middleware": "http://localhost/middleware.php",
          "aggmode": "none",
          "duplicates": 0
        },
        {
          "api": "volkszaehler",
          "uuid": "29062040-cee9-11ec-a869-8741266b2e40",
          "identifier": "temperature_dew_point_mean_200",
          "middleware": "http://localhost/middleware.php",
          "aggmode": "none",
          "duplicates": 0
        }
      ],
      "protocol": "file",
      "path": "/home/pi/solar/dwd_daten.dat",
      "format": "$t $i $v"
    }

This is the log file. Because it reads the first 32 lines again, they are duplicates now. In the first run they have been added. If I delete the first lines from the file, the next ones are added.

[May 10 00:21:41][main] vzlogger v0.8.1 based on heads/master-0-g76e868feed from Wed, 20 Apr 2022 08:39:54 +0200 started.
[May 10 00:21:41][push] Ignoring push entry due to empty array or duplicate section
[May 10 00:21:41][mtr0] Creating new meter with protocol exec.
[May 10 00:21:41][exec] MeterExec::MeterExec: Parsed format string "$t $i $v" => "%3$lf %2$ms %1$lf"
[May 10 00:21:41][mtr0] Meter configured, disabled.
[May 10 00:21:41]       New meter initialized (protocol=exec)
[May 10 00:21:41]       Configure channel.
[May 10 00:21:41][chn0] New channel initialized (uuid=...d81f4c api=volkszaehler id=Altitude)
[May 10 00:21:41]       Configure channel.
[May 10 00:21:41][chn1] New channel initialized (uuid=...2c38c5 api=volkszaehler id=Azimut)
[May 10 00:21:41]       Configure channel.
[May 10 00:21:41][chn2] New channel initialized (uuid=...688e1b api=volkszaehler id=Radiation)
[May 10 00:21:41][mtr1] Creating new meter with protocol file.
[May 10 00:21:41][file] Parsed format string "$t $i $v" => "%3$lf %2$ms %1$lf"
[May 10 00:21:41][mtr1] Meter configured, enabled.
[May 10 00:21:41]       New meter initialized (protocol=file)
[May 10 00:21:41]       Configure channel.
[May 10 00:21:41][chn3] New channel initialized (uuid=...0e7bf9 api=volkszaehler id=radiation_global)
[May 10 00:21:41]       Configure channel.
[May 10 00:21:41][chn4] New channel initialized (uuid=...6b2e40 api=volkszaehler id=temperature_dew_point_mean_200)
[May 10 00:21:41]       Have 2 meters.
[May 10 00:21:41][main] log level is 10
[May 10 00:21:41][main] local=1
[May 10 00:21:41]       Daemonize process...
[May 10 00:21:41]       Opened logfile /var/log/vzlogger.log
[May 10 00:21:41][]     ===> Start meters
[May 10 00:21:41][mtr0] Meter for protocol 'exec' is disabled. Skipping.
[May 10 00:21:41][mtr1] Meter connection established
[May 10 00:21:41][mtr1] Meter thread started
[May 10 00:21:41][mtr1] Meter is opened. Starting channels.
[May 10 00:21:41][chn3] Logging thread started
[May 10 00:21:41][chn4] Logging thread started
[May 10 00:21:41][http] Starting local interface HTTPd on port 8080
[May 10 00:21:41][]     Startup done.
[May 10 00:21:41][chn3] Start logging thread for volkszaehler-api.
[May 10 00:21:41][chn3] Using default volkszaehler api.
[May 10 00:21:41][chn4] Start logging thread for volkszaehler-api.
[May 10 00:21:41][chn4] Using default volkszaehler api.
[May 10 00:21:41][mtr1] Number of readers: 32
[May 10 00:21:41][mtr1] Config.local: 1
[May 10 00:21:41][]     MeterFile::read: 32, 32
[May 10 00:21:41][]     MeterFile::read: '1651717200               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651717200.000000
[May 10 00:21:41][]     MeterFile::read: '1651717800               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651717800.000000
[May 10 00:21:41][]     MeterFile::read: '1651718400               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651718400.000000
[May 10 00:21:41][]     MeterFile::read: '1651719000               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651719000.000000
[May 10 00:21:41][]     MeterFile::read: '1651719600               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651719600.000000
[May 10 00:21:41][]     MeterFile::read: '1651720200               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651720200.000000
[May 10 00:21:41][]     MeterFile::read: '1651720800               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651720800.000000
[May 10 00:21:41][]     MeterFile::read: '1651721400               radiation_global      0.0'
[May 10 00:21:41][]     MeterFile::read: 0.000000, radiation_global, 1651721400.000000
[May 10 00:21:41][]     MeterFile::read: '1651722000               radiation_global   1000.0'
[May 10 00:21:41][]     MeterFile::read: 1000.000000, radiation_global, 1651722000.000000
[May 10 00:21:41][]     MeterFile::read: '1651722600               radiation_global   3000.0'
[May 10 00:21:41][]     MeterFile::read: 3000.000000, radiation_global, 1651722600.000000
[May 10 00:21:41][]     MeterFile::read: '1651723200               radiation_global   8000.0'
[May 10 00:21:41][]     MeterFile::read: 8000.000000, radiation_global, 1651723200.000000
[May 10 00:21:41][]     MeterFile::read: '1651723800               radiation_global  15000.0'
[May 10 00:21:41][]     MeterFile::read: 15000.000000, radiation_global, 1651723800.000000
[May 10 00:21:41][]     MeterFile::read: '1651724400               radiation_global  21000.0'
[May 10 00:21:41][]     MeterFile::read: 21000.000000, radiation_global, 1651724400.000000
[May 10 00:21:41][]     MeterFile::read: '1651725000               radiation_global  27000.0'
[May 10 00:21:41][]     MeterFile::read: 27000.000000, radiation_global, 1651725000.000000
[May 10 00:21:41][]     MeterFile::read: '1651725600               radiation_global  40000.0'
[May 10 00:21:41][]     MeterFile::read: 40000.000000, radiation_global, 1651725600.000000
[May 10 00:21:41][]     MeterFile::read: '1651726200               radiation_global  55000.0'
[May 10 00:21:41][]     MeterFile::read: 55000.000000, radiation_global, 1651726200.000000
[May 10 00:21:41][]     MeterFile::read: '1651726800               radiation_global  70000.0'
[May 10 00:21:41][]     MeterFile::read: 70000.000000, radiation_global, 1651726800.000000
[May 10 00:21:41][]     MeterFile::read: '1651727400               radiation_global  58000.0'
[May 10 00:21:41][]     MeterFile::read: 58000.000000, radiation_global, 1651727400.000000
[May 10 00:21:41][]     MeterFile::read: '1651728000               radiation_global  95000.0'
[May 10 00:21:41][]     MeterFile::read: 95000.000000, radiation_global, 1651728000.000000
[May 10 00:21:41][]     MeterFile::read: '1651728600               radiation_global 127000.0'
[May 10 00:21:41][]     MeterFile::read: 127000.000000, radiation_global, 1651728600.000000
[May 10 00:21:41][]     MeterFile::read: '1651729200               radiation_global 101000.0'
[May 10 00:21:41][]     MeterFile::read: 101000.000000, radiation_global, 1651729200.000000
[May 10 00:21:41][]     MeterFile::read: '1651729800               radiation_global 122000.0'
[May 10 00:21:41][]     MeterFile::read: 122000.000000, radiation_global, 1651729800.000000
[May 10 00:21:41][]     MeterFile::read: '1651730400               radiation_global 158000.0'
[May 10 00:21:41][]     MeterFile::read: 158000.000000, radiation_global, 1651730400.000000
[May 10 00:21:41][]     MeterFile::read: '1651731000               radiation_global 172000.0'
[May 10 00:21:41][]     MeterFile::read: 172000.000000, radiation_global, 1651731000.000000
[May 10 00:21:41][]     MeterFile::read: '1651731600               radiation_global 215000.0'
[May 10 00:21:41][]     MeterFile::read: 215000.000000, radiation_global, 1651731600.000000
[May 10 00:21:41][]     MeterFile::read: '1651732200               radiation_global 210000.0'
[May 10 00:21:41][]     MeterFile::read: 210000.000000, radiation_global, 1651732200.000000
[May 10 00:21:41][]     MeterFile::read: '1651732800               radiation_global 191000.0'
[May 10 00:21:41][]     MeterFile::read: 191000.000000, radiation_global, 1651732800.000000
[May 10 00:21:41][]     MeterFile::read: '1651733400               radiation_global 199000.0'
[May 10 00:21:41][]     MeterFile::read: 199000.000000, radiation_global, 1651733400.000000
[May 10 00:21:41][]     MeterFile::read: '1651734000               radiation_global 170000.0'
[May 10 00:21:41][]     MeterFile::read: 170000.000000, radiation_global, 1651734000.000000
[May 10 00:21:41][]     MeterFile::read: '1651734600               radiation_global 181000.0'
[May 10 00:21:41][]     MeterFile::read: 181000.000000, radiation_global, 1651734600.000000
[May 10 00:21:41][]     MeterFile::read: '1651735200               radiation_global 207000.0'
[May 10 00:21:41][]     MeterFile::read: 207000.000000, radiation_global, 1651735200.000000
[May 10 00:21:41][]     MeterFile::read: '1651735800               radiation_global 208000.0'
[May 10 00:21:41][]     MeterFile::read: 208000.000000, radiation_global, 1651735800.000000
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651717200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651717800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651718400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651719000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651719600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651720200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651720800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=0.00 ts=1651721400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=1000.00 ts=1651722000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=3000.00 ts=1651722600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=8000.00 ts=1651723200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=15000.00 ts=1651723800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=21000.00 ts=1651724400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=27000.00 ts=1651725000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=40000.00 ts=1651725600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=55000.00 ts=1651726200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=70000.00 ts=1651726800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=58000.00 ts=1651727400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=95000.00 ts=1651728000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=127000.00 ts=1651728600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=101000.00 ts=1651729200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=122000.00 ts=1651729800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=158000.00 ts=1651730400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=172000.00 ts=1651731000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=215000.00 ts=1651731600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=210000.00 ts=1651732200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=191000.00 ts=1651732800000)
[May 10 00:21:41][chn3] Adding reading to queue (value=199000.00 ts=1651733400000)
[May 10 00:21:41][chn3] Adding reading to queue (value=170000.00 ts=1651734000000)
[May 10 00:21:41][chn3] Adding reading to queue (value=181000.00 ts=1651734600000)
[May 10 00:21:41][chn3] Adding reading to queue (value=207000.00 ts=1651735200000)
[May 10 00:21:41][chn3] Adding reading to queue (value=208000.00 ts=1651735800000)
[May 10 00:21:41][chn3] ==> number of tuples: 32
[May 10 00:21:41][chn3] compare: 0 1651717200000
[May 10 00:21:41][chn3] compare: 1651717200000 1651717800000
[May 10 00:21:41][chn3] compare: 1651717800000 1651718400000
[May 10 00:21:41][chn3] compare: 1651718400000 1651719000000
[May 10 00:21:41][chn3] compare: 1651719000000 1651719600000
[May 10 00:21:41][chn3] compare: 1651719600000 1651720200000
[May 10 00:21:41][chn3] compare: 1651720200000 1651720800000
[May 10 00:21:41][chn3] compare: 1651720800000 1651721400000
[May 10 00:21:41][chn3] compare: 1651721400000 1651722000000
[May 10 00:21:41][chn3] compare: 1651722000000 1651722600000
[May 10 00:21:41][chn3] compare: 1651722600000 1651723200000
[May 10 00:21:41][chn3] compare: 1651723200000 1651723800000
[May 10 00:21:41][chn3] compare: 1651723800000 1651724400000
[May 10 00:21:41][chn4] ==> number of tuples: 0
[May 10 00:21:41][chn4] JSON request body is null. Nothing to send now.
[May 10 00:21:41][mtr1] waiting 600 seconds before next reading
[May 10 00:21:41][chn3] compare: 1651724400000 1651725000000
[May 10 00:21:41][chn3] compare: 1651725000000 1651725600000
[May 10 00:21:41][chn3] compare: 1651725600000 1651726200000
[May 10 00:21:41][chn3] compare: 1651726200000 1651726800000
[May 10 00:21:41][chn3] compare: 1651726800000 1651727400000
[May 10 00:21:41][chn3] compare: 1651727400000 1651728000000
[May 10 00:21:41][chn3] compare: 1651728000000 1651728600000
[May 10 00:21:41][chn3] compare: 1651728600000 1651729200000
[May 10 00:21:41][chn3] compare: 1651729200000 1651729800000
[May 10 00:21:41][chn3] compare: 1651729800000 1651730400000
[May 10 00:21:41][chn3] compare: 1651730400000 1651731000000
[May 10 00:21:41][chn3] compare: 1651731000000 1651731600000
[May 10 00:21:41][chn3] compare: 1651731600000 1651732200000
[May 10 00:21:41][chn3] compare: 1651732200000 1651732800000
[May 10 00:21:41][chn3] compare: 1651732800000 1651733400000
[May 10 00:21:41][chn3] compare: 1651733400000 1651734000000
[May 10 00:21:41][chn3] compare: 1651734000000 1651734600000
[May 10 00:21:41][chn3] compare: 1651734600000 1651735200000
[May 10 00:21:41][chn3] compare: 1651735200000 1651735800000
[May 10 00:21:41][chn3] JSON request body: [ [ 1651717200000, 0 ], [ 1651717800000, 0 ], [ 1651718400000, 0 ], [ 1651719000000, 0 ], [ 1651719600000, 0 ], [ 1651720200000, 0 ], [ 1651720800000, 0 ], [ 1651721400000, 0 ], [ 1651722000000, 1000 ], [ 1651722600000, 3000 ], [ 1651723200000, 8000 ], [ 1651723800000, 15000 ], [ 1651724400000, 21000 ], [ 1651725000000, 27000 ], [ 1651725600000, 40000 ], [ 1651726200000, 55000 ], [ 1651726800000, 70000 ], [ 1651727400000, 58000 ], [ 1651728000000, 95000 ], [ 1651728600000, 127000 ], [ 1651729200000, 101000 ], [ 1651729800000, 122000 ], [ 1651730400000, 158000 ], [ 1651731000000, 172000 ], [ 1651731600000, 215000 ], [ 1651732200000, 210000 ], [ 1651732800000, 191000 ], [ 1651733400000, 199000 ], [ 1651734000000, 170000 ], [ 1651734600000, 181000 ], [ 1651735200000, 207000 ], [ 1651735800000, 208000 ] ]
[May 10 00:21:41][chn3] Middleware says duplicated value. Removing first entry!
[May 10 00:21:41][chn3] CURL Error from middleware: 'UniqueConstraintViolationException': 'An exception occurred while executing 'INSERT INTO data (channel_id, timestamp, value) VALUES (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?,?), (4,?
[May 10 00:21:41][chn3] Waiting 0 secs for next request due to previous failure

Is there are way to make vzlogger read the whole file? I don't want to execute the python script many time because it takes some time (which is mostly overhead and not dependent on the amount of data that I download).

Thanks!

drtorchwood avatar May 09 '22 22:05 drtorchwood

I had it running now for a few days. It turned out that it is skipping the one further line with every call until the end of the file is reached. However, it never starts reading from the beginning again after reading the whole file.

drtorchwood avatar May 16 '22 20:05 drtorchwood

looking at the source: https://github.com/volkszaehler/vzlogger/blob/76e868feede73b3f86b63d031768f71a227a903c/src/protocols/MeterFile.cpp#L239 https://github.com/volkszaehler/vzlogger/blob/76e868feede73b3f86b63d031768f71a227a903c/src/threads.cpp#L78 https://github.com/volkszaehler/vzlogger/blob/76e868feede73b3f86b63d031768f71a227a903c/src/Meter.cpp#L58 note that n=max_readings=32, the number on the line:

METER_DETAIL(file, File, "Read from file or fifo", 32),

the vzlogger core accepts a fixed maximum of readings from a meter per call to meter->read(), which is set to 32 for MeterFile. you could increase that and recompile, but note that MeterFile was never intended to import data dumps, it ahould probably be adjusted to remember the position in the file and return further readings in the next meter->read() call, instead of resetting to the beginning of the file. it should probably at least log a wsrning if max_readings is exceeded.

r00t- avatar Jun 10 '22 20:06 r00t-