neowx-material
neowx-material copied to clipboard
neoground
Feature/history report
Added the history report from https://github.com/ngulden/niculskin/ and took also some version updates from dependabot
Nice add on - how about wind? Average and max speeds?
Hi @uktricky
I added wind tables... please have a look
https://solar.seehausen.org/weewx/history.html
Looks good, I've just gotten around to looking this again, work always getting in the way! I've created a dev version of my current NeoWX-Material installation so I can have a play in dev before I risk breaking my live site.
Might be a daft question but I am not an expert when using GitHub by any means! How do I get your history pages into my version? Would you mind sending me some guidance? Thanks in advance...
Looks good, I've just gotten around to looking this again, work always getting in the way! I've created a dev version of my current NeoWX-Material installation so I can have a play in dev before I risk breaking my live site.
Might be a daft question but I am not an expert when using GitHub by any means! How do I get your history pages into my version? Would you mind sending me some guidance? Thanks in advance...
How to get History Reports download the files from https://github.com/seehase/neowx-material/archive/refs/heads/feature/history-report.zip
Extract ZIP file and copy these files
src/footer.inc
src/header.inc
src/history.html.tmpl
src/skin.conf
to folder
/etc/weewx/skins/neowx-material
Copy
bin/user/historygenerator.py
to
/usr/share/weewx/user
then restart weewx and you should see the History Report after the next generation run
@seehase Is this a render issue on Safari? Or a small bug? This depends on the screen resolution I'm not a Frontend expert, probably this can be optimised ;-), maybe you can help? but the colors looks different to what I have
Yeah, it seems it like a resolution thing. I'm on a newer MacBook with pretty big resolution. If I shrink the browser window, it seems ok.
Looks good, I've just gotten around to looking this again, work always getting in the way! I've created a dev version of my current NeoWX-Material installation so I can have a play in dev before I risk breaking my live site. Might be a daft question but I am not an expert when using GitHub by any means! How do I get your history pages into my version? Would you mind sending me some guidance? Thanks in advance...
How to get History Reports download the files from https://github.com/seehase/neowx-material/archive/refs/heads/feature/history-report.zip
Extract ZIP file and copy these files
src/footer.inc src/header.inc src/history.html.tmpl src/skin.confto folder
/etc/weewx/skins/neowx-materialCopy
bin/user/historygenerator.pyto
/usr/share/weewx/userthen restart weewx and you should see the History Report after the next generation run
Many thanks - I did find the files you'd changed and hacked around so got it working, had to amend the skin.conf as I'd changed the original from the original release. Thanks again for pulling this feature together be good to see it merged into the release version.
Re. the other posts on this thread, I'm running on Safari also but again larger resolution.
So I tried it a third time. This time I deleted the "old" neowx-material folder and used it from the uploaded zip file.
Still the same issues. Rendering is off and just the first value got a css background. The rendering is on @seehase Site off too. So this does not seems a problem on my side directly.
There is also a thread on the wetterstationsforum.info forum with a small manual bit with slightly other instructions. I also tried this, but without success.
So I tried it a third time. This time I deleted the "old" neowx-material folder and used it from the uploaded zip file.
Still the same issues. Rendering is off and just the first value got a css background. The rendering is on @seehase Site off too. So this does not seems a problem on my side directly.
There is also a thread on the wetterstationsforum.info forum with a small manual bit with slightly other instructions. I also tried this, but without success.
I know, it's because of the responsive design, and breakpoints for specific screen resolutions. I'll try to fix it, or just prevent switching
So I tried it a third time. This time I deleted the "old" neowx-material folder and used it from the uploaded zip file.
Still the same issues. Rendering is off and just the first value got a css background. The rendering is on @seehase Site off too. So this does not seems a problem on my side directly.
There is also a thread on the wetterstationsforum.info forum with a small manual bit with slightly other instructions. I also tried this, but without success.
I fixed it somehow. not yet perfect but it does not break anymore
just take the history.html.tmpl file from last commit...
Maybe we find some Bootstrap guru who can solve it finally
Thank you @seehase for this improvement. My use of US units seems to be causing trouble, particularly with parsing rainfall data which is stored in the the non-integer #.## format. When I first merged your code and added the [HistoryReport] section to skins conf, I used #.## formatted values for minvalues and maxvalues under the [[rain]] subsection. For example:
#minvalues = 0.00, 0.10, 0.25, 0.50, 0.75, 1.00 # Override default temperature colour scheme with rain specific scale 360 #maxvalues = 0.10, 0.25, 0.50, 0.70, 1.00, 10.00
When I ran wee_reports using those values, I received this error:
pi@weewx: /etc/weewx/skins/neowx-material $ sudo wee_reports Using configuration file /etc/weewx/weewx.conf Generating for all time Traceback (most recent call last): File "/usr/share/weewx/weewx/reportengine.py", line 197, in run obj.start() File "/usr/share/weewx/weewx/reportengine.py", line 385, in start self.run() File "/usr/share/weewx/weewx/cheetahgenerator.py", line 169, in run ngen = self.generate(gen_dict[section_name], section_name, self.gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 229, in generate ngen += self.generate(section[subsection], subsection, gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 229, in generate ngen += self.generate(section[subsection], subsection, gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 321, in generate searchList = self._getSearchList(encoding, timespan, File "/usr/share/weewx/weewx/cheetahgenerator.py", line 415, in _getSearchList search_list += obj.get_extension_list(timespan, db_lookup) File "/usr/share/weewx/user/historygenerator.py", line 171, in get_extension_list self.search_list_extension[table_name] = self._statsHTMLTable(table_options, table_stats, table_name, File "/usr/share/weewx/user/historygenerator.py", line 291, in _statsHTMLTable htmlLine += (' ' * 12) + self._colorCell(value[0], format_string, bgColours) File "/usr/share/weewx/user/historygenerator.py", line 317, in _colorCell if (value >= int(c[0])) and (value <= int(c[1])): ValueError: invalid literal for int() with base 10: '0.00'
It seemed the code was not handling the floats, so I modified skins.conf to use integers for minvalues and maxvalues as a temporary workaround and ran wee_reports again. The config file modification addressed the original error, but this time I received a different error:
pi@weewx: /usr/share/weewx/user $ sudo wee_reports Using configuration file /etc/weewx/weewx.conf Generating for all time Traceback (most recent call last): File "/usr/share/weewx/weewx/reportengine.py", line 197, in run obj.start() File "/usr/share/weewx/weewx/reportengine.py", line 385, in start self.run() File "/usr/share/weewx/weewx/cheetahgenerator.py", line 169, in run ngen = self.generate(gen_dict[section_name], section_name, self.gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 229, in generate ngen += self.generate(section[subsection], subsection, gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 229, in generate ngen += self.generate(section[subsection], subsection, gen_ts) File "/usr/share/weewx/weewx/cheetahgenerator.py", line 321, in generate searchList = self._getSearchList(encoding, timespan, File "/usr/share/weewx/weewx/cheetahgenerator.py", line 415, in _getSearchList search_list += obj.get_extension_list(timespan, db_lookup) File "/usr/share/weewx/user/historygenerator.py", line 171, in get_extension_list self.search_list_extension[table_name] = self._statsHTMLTable(table_options, table_stats, table_name, File "/usr/share/weewx/user/historygenerator.py", line 227, in _statsHTMLTable unit_type = reading.converter.group_unit_dict[reading.value_t[2]] File "/usr/share/weewx/weewx/tags.py", line 488, in getattr vh = self._do_query() File "/usr/share/weewx/weewx/tags.py", line 474, in _do_query result = weewx.xtypes.get_aggregate(self.obs_type, self.timespan, File "/usr/share/weewx/weewx/xtypes.py", line 127, in get_aggregate return xtype.get_aggregate(obs_type, timespan, aggregate_type, db_manager, File "/usr/share/weewx/weewx/xtypes.py", line 455, in get_aggregate target_val = weewx.units.convertStd(val, db_manager.std_unit_system)[0] File "/usr/share/weewx/weewx/units.py", line 1444, in convertStd return StdUnitConverters[target_std_unit_system].convert(val_t) File "/usr/share/weewx/weewx/units.py", line 879, in convert new_unit_type = self.group_unit_dict.get(val_t[2], USUnits[val_t[2]]) IndexError: tuple index out of range
This second error seems to occur during the conversion to US units, and I have never received it before when generating reports. I presume my weewx database contains rain data stored in non-integer format, but I don't have the python knowledge to determine whether that is resulting in this error. I would also like to be able to use appropriate US non-integer units for minunits and maxunits in skins.conf.
Can you help?
HI @ferretstew database values in non-integer is no problem, they are always stored in numeric
To have min/max ranges in numeric you can take my latest version or just change this line https://github.com/neoground/neowx-material/blob/bdf74c433ea8f03fc6622599e7ff2e805b7261bf/bin/user/historygenerator.py#L317 of historygenerator.py to
if (value >= float(c[0])) and (value <= float(c[1])):
and restart weewx service
float value can be set in skin.conf like
minvalues = 0.00, 1.10, 3.25, 7.50, 10.75, 12.10
maxvalues = 1.10, 3.25, 7.50, 10.75, 12.10, 100.40
for your problem with us-units, please send me your wewwx.conf and skin.conf, then I can have a look, It will take a while because I will be on vacation
I made the recommended change to historygenerator.py and restarted weewx, but I still receive the second error mentioned in my last post ("tuple index out of range" on line 879 of units.py).
I have attached a zipfile of my weewx.conf and skin.conf, modified to obscure passwords. I don't think I've modified skin.conf. I believe all mods I've made the skin configuration are contained in my weewx.conf. Also, you will notice I've reverted to using floats in my rain minvalues and maxvalues (for example, "0.10").
Thank you!
Hi ferretstew US Units should not be a problem see https://solar.seehausen.org/weewx/index.html I currently changed the units to US
I think the problem is your configuration, it is a bit "mixed" Best solution would be start a new configuration from scratch, and configure your driver and parameter have a look into my config files, e.g. HistoryReport config should be in the skin.conf
you can take the attached skin.conf without any changes, only your weewx.conf should be adapted take the default weewx conf and make the changes for your driver, and skin, then it should work
if you have a question please send me a mail to [email protected]
This is great, I have added it manually to my site as I have so many mods to the skin, header and footer files. I don't seem to be seeing colours on all the values, is it because I don't have a full year of data, or is there something amiss?
In the syslog I get this message which may be a clue: No bootstrap specific labels found.
Thanks
This is great, I have added it manually to my site as I have so many mods to the skin, header and footer files. I don't seem to be seeing colours on all the values, is it because I don't have a full year of data, or is there something amiss?
Hi @Pogs2004
I think it is not a problem of missing values. I could imagine that minvalue/maxvalue/colours do not match. Defaults are defined in
[HistoryReport]
minvalues = -50, -10, -5, 0, 5, 10, 15, 20, 25, 30, 35
maxvalues = -10, -5, 0, 5, 10, 15, 20, 25, 30, 35, 60
colours = "#0029E5", "#0186E7", "#02E3EA", "#04EC97", "#05EF3D2", "#2BF207", "#8AF408", "#E9F70A", "#F9A90B", "#FC4D0D", "#FF0F2D"
but they can be overwritten for each section e.g. if you have only small values like "Minimum Temperature" you should use smaller ranges like I see value between -0.8 ... 8.0
[min_temp]
minvalues = -3, 0, 3, 5, 7, 9
maxvalues = 0, 3, 5, 7, 9, 11
colours = "#0029E5", "#0186E7", "#02E3EA", "#04EC97", "#05EF3D2", "#2BF207"
in your "Rainfall" you have values from 25.2 to 136.5 so here you need to choose bigger interval Just take the hole section of [HistoryReport] and play around with the values
This is great, I have added it manually to my site as I have so many mods to the skin, header and footer files. I don't seem to be seeing colours on all the values, is it because I don't have a full year of data, or is there something amiss?
Hi @Pogs2004
I think it is not a problem of missing values. I could imagine that minvalue/maxvalue/colours do not match. Defaults are defined in
[HistoryReport] minvalues = -50, -10, -5, 0, 5, 10, 15, 20, 25, 30, 35 maxvalues = -10, -5, 0, 5, 10, 15, 20, 25, 30, 35, 60 colours = "#0029E5", "#0186E7", "#02E3EA", "#04EC97", "#05EF3D2", "#2BF207", "#8AF408", "#E9F70A", "#F9A90B", "#FC4D0D", "#FF0F2D"but they can be overwritten for each section e.g. if you have only small values like "Minimum Temperature" you should use smaller ranges like I see value between -0.8 ... 8.0
[min_temp] minvalues = -3, 0, 3, 5, 7, 9 maxvalues = 0, 3, 5, 7, 9, 11 colours = "#0029E5", "#0186E7", "#02E3EA", "#04EC97", "#05EF3D2", "#2BF207"in your "Rainfall" you have values from 25.2 to 136.5 so here you need to choose bigger interval Just take the hole section of [HistoryReport] and play around with the values
Thanks @seehase, I have updated the ranges to suit my data but it is still only colouring the first cell in each card. I notice that I get an error in the syslog:
Oct 25 22:10:18 weather weewxd: historygenerator.py: No bootstrap specific labels found
Oct 25 22:10:18 weather weewxd: Generated table min_temp_table
Oct 25 22:10:18 weather weewxd: Generated table max_temp_table
Oct 25 22:10:18 weather weewxd: Generated table avg_temp_table
Oct 25 22:10:18 weather weewxd: Generated table rain_table
Oct 25 22:10:18 weather weewxd: Generated table rain_days_table
Oct 25 22:10:18 weather weewxd: Generated table uv_max_table
Oct 25 22:10:18 weather weewxd: Generated table max_wind_table
Oct 25 22:10:18 weather weewxd: Generated table avg_wind_table
Oct 25 22:10:18 weather weewxd: historygenerator.py: Generated 8 tables in 0.04 seconds
Could the bootstrap labels have anything to do with it? The generated history.html file is a bit broken, the first table data cell in each row (the only cell with colour) has several bgcolor tags and the remaining cells just have the value, no colour tags.
Thanks!
I don't think it is the missing label. It must be something with the range configuration the method looks straight forward Maybe wrong delimiters? there have to be 3 lists [minvalues], [maxvalues], [html colour code] maybe they are not correctly inititalized? I would expect if there is a problem while cast to flow that you see another exception
def _colorCell(self, value, format_string, bgColours):
"""Returns a '<td> bgcolor = xxx y.yy </td>' html table entry string.
value: Numeric value for the observation
format_string: How the numberic value should be represented in the table cell.
bgColours: An array containing 3 lists. [minvalues], [maxvalues], [html colour code]
"""
if value is not None:
cellText = "<td"
for c in bgColours:
if (value >= float(c[0])) and (value <= float(c[1])):
cellText += " bgcolor = \"%s\"" % c[2]
formatted_value = format_string % value
cellText += "> %s </td>" % formatted_value
else:
cellText = "<td>-</td>\n"
return cellText
@seehase I have checked all the lists and values and they all seem fine, exactly the same number of min values, max values and colours in each series. The history html tables look like this:
<table class="table"> <thead> <tr> <th>°C</th> <th>Jan</th> <th>Feb</th> <th>Mar</th> <th>Apr</th> <th>May</th> <th>Jun</th> <th>Jul</th> <th>Aug</th> <th>Sep</th> <th>Oct</th> <th>Nov</th> <th>Dec</th> </tr> </thead> <tbody> <tr>
<td>2022</td>
<td>-</td>
<td bgcolor = "#0029E5" bgcolor = "#0186E7" bgcolor = "#02E3EA" bgcolor = "#04EC97"> -0.8 </td> <td> -0.2 </td> <td> -0.4 </td> <td> 3.2 </td> <td> 6.6 </td> <td> 7.7 </td> <td> 8.0 </td> <td> 4.6 </td> <td> 3.5 </td> <td>-</td>
<td>-</td>
</tr>
</tr>
</tbody>
</table>
You can see it the first
It should look like this
<td>2018</td>
--
| <td bgcolor = "#02E3EA"> -1.1 </td> <td bgcolor = "#0029E5"> -11.5 </td> <td bgcolor = "#0186E7"> -8.9 </td> <td bgcolor = "#04EC97"> 2.9 </td>
looks like there is a missing or wrong delimiter in the colourCode list. must be separated by "," and you can try a single quote ' instead of " and also check for tab's or other whitespaces? just a guess?
It should look like this
<td>2018</td> -- | <td bgcolor = "#02E3EA"> -1.1 </td> <td bgcolor = "#0029E5"> -11.5 </td> <td bgcolor = "#0186E7"> -8.9 </td> <td bgcolor = "#04EC97"> 2.9 </td>looks like there is a missing or wrong delimiter in the colourCode list. must be separated by "," and you can try a single quote ' instead of " and also check for tab's or other whitespaces? just a guess?
Not sure what's going on. I have copied back the original [HistoryReport] section from this branch and still get the same. I edited my version using ' instead of " and made sure all the values are correct but no change. As you say, it's likely a typo somewhere but I can't find it!
@cbirkenbeul I notice you had the same issue with only the first cells coloured, did you ever find out what was causing it?
you can send my your skin.conf and I can check... maybe the error is somewhere else e.g. wewwx.conf I don't know
you can send my your skin.conf and I can check... maybe the error is somewhere else e.g. wewwx.conf I don't know
Here's my skin.conf, you can see I have edited the [HistoryGenerator] section, this one I have replaced " with ' and also put the min, max and colour tags in each group rather than default - this made no difference!
The only addition to my weewx.conf is the historygenerator.py line, I can see nothing in there that could be causing this as I keep all the skin configs in their own folder.
I def. support this PR. I've had the history reports setup on my own instance since neowx-material was released (https://wx.w0chp.net/stats/); because I had the reports on my prior skin and wanted to retain it.
However, I see folks struggling to get it working. If anyone needs code snippets/configs/etc., I'd be happy to provide them.
Strange ... your skin.conf works for me
https://solar.seehausen.org/weewx/history.html
What OS are you using? Linux, Win? Could it be problems with file encoding? UTF8, UNIX/DOS?
try delete the file /usr/share/weewx/user/historygenerator.pyc
and copy latest historygenerator.py from my PR and restart weewx
Thanks again @seehase, I really appreciate your help. I’ll update the historygenerator.py when I’m back at my laptop. Weewx is running on a Raspberry Pi 4 / Raspberry Pi OS Lite 64bit with UTF8 encoding, as you can see the rest of Neowx-Material is working fine, I have some hard coded config changes in the header and footer.inc files, and most of the *.tmpl files have mods, but not history.html.tmpl, it’s vanilla. The weewx.conf file is fairly standard apart from some changed QC values, some calculate values and it is using an extension driver, weatherlinklive - weewx gets its data from a Davis WLL via ethernet (my weather station sits in its own subnet on my network).
I’ll let you know how I get on, now I know it’s not the skin.conf I can look elsewhere!
I def. support this PR. I've had the history reports setup on my own instance since
neowx-materialwas released (https://wx.w0chp.net/stats/); because I had the reports on my prior skin and wanted to retain it.However, I see folks struggling to get it working. If anyone needs code snippets/configs/etc., I'd be happy to provide them.
Thanks @W0CHP, Seehase has been helping out but another pair of eyes on the problem would be good! I have the history page up and running, however only the first cell in each card is coloured: https://www.360shetland.co.uk/weather/history.html
Seehase used my skin.conf on his build and it worked fine, I am using the latest historygenerator.py so not sure what's going on! The rendered history.html has the first
Hi @Pogs2004 , here are the relevant colour (etc.) values of my skin.conf's [HistoryReport] stanza:
[HistoryReport]
minvalues = -65, -10, 0, 5, 10, 15, 25, 35, 45, 50, 60
maxvalues = -5, 0, 10, 20, 35, 45, 60, 75, 90, 100, 120
colours = "#0098F6", "#0098F6", "#4FBCFF", "#0bdcd0", "#5EF3D2", "#2BF207", "#8AF408", "#E9F70A", "#F9A90B", "#FC4D0D", "#FF0F2D"
monthnames = Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec
[[max_temp]]
obs_type = outTemp
aggregate_type = max
[[avg_temp]]
obs_type = outTemp
aggregate_type = avg
minvalues = -45, -10, 0, 5, 10, 15, 25, 35, 45, 50, 60
maxvalues = -5, 0, 10, 20, 35, 45, 60, 70, 85, 100, 120
colours = "#0186E7", "#0186E7", "#02E3EA", "#04EC97", "#5EF3D2", "#2BF207", "#8AF408", "#E9F70A", "#F9A90B", "#FC4D0D", "#FF0F2D"
[[avg_htemp]]
obs_type = inTemp
aggregate_type = avg
minvalues = 50, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80
maxvalues = 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 85
[[temp_count]]
obs_type = outTemp
aggregate_type = sum_ge
aggregate_threshold = -150.0, degree_F
minvalues = 0, 2, 4, 6, 8, 12, 15, 18, 21, 24, 27
maxvalues = 2, 4, 6, 8, 12, 15, 18, 21, 24, 27, 32
colours = "#0029E5", "#0186E7", "#02E3EA", "#04EC97", "#05EF3D2", "#2BF207", "#8AF408", "#E9F70A", "#F9A90B", "#FC4D0D", "#FFFFFF"
[[avg_wind]]
obs_type = wind
aggregate_type = avg
minvalues = 0, 1, 2, 3, 4, 5
maxvalues = 1, 3, 5, 7, 9, 20
colours = '#f6e8f5', '#edd1eb', '#e4bae1', '#dba3d7', '#d38ccd', '#ca75c2'
[[max_wind]]
obs_type = wind
aggregate_type = max
minvalues = 0, 5, 10, 15, 20, 25, 30, 35, 40, 45
maxvalues = 10, 20, 25, 30, 40, 45, 50, 60, 75, 100
colours = '#f6e8f5', '#edd1eb', '#e4bae1', '#dba3d7', '#d38ccd', '#ca75c2', '#c15eb8', '#b847ae', '#af30a4', '#a6199a'