Powerwall-Dashboard icon indicating copy to clipboard operation
Powerwall-Dashboard copied to clipboard

Repo cleanup.. remove ignored files, update ignore list

Open YesThatAllen opened this issue 1 year ago • 29 comments

this is an offer for me to help, I'm looking for confirmation & maybe some guidance

Problem

I'm seeing a number of changes in my local git gui which could be prevented with some cleanup.

Enhancement

  • There are some .DS_Store files in the repo which are ignored & show as diffs
  • it looks like there are some database files in the repo which need cleanup
  • adding some well placed .keep files can ensure folders are in place upon git checkout
  • I see some changes in the backup script I'd submit in a separate PR once this is done.

Additional context

I've been running cross platform dev teams for a while. Not everyone knows about a .gitattributes file or .editorconfig. I'm not super familiar with how to help the actionable code, but I can sure clean up a repo. 😄

I plan on opening a PR with changes soon, I'll have some questions about how best stuff should work.

My local diff in the moment:

2023-05-02_10-12-57

YesThatAllen avatar May 02 '23 15:05 YesThatAllen

Hi @YesThatAllen - I would love a PR for this! Thank for your help. 🙏

jasonacox avatar May 03 '23 03:05 jasonacox

Here's a start to the work.. all feedback is welcome.

I've got a question regarding timezones.. I wonder if the diff I'm about to paste in could be handled in a separate issue regarding timezone support, but maybe I'm setting my tz to Chicago the wrong way, resulting in the diff.

I'm mostly curious about the change in the /influxdb/influxdb.sql file

TZ related diff

I still have this diff after running the update script

% git  --no-pager diff
diff --git a/dashboard.json b/dashboard.json
index b09a845..cadbd04 100644
--- a/dashboard.json
+++ b/dashboard.json
@@ -34,7 +34,7 @@
       "name": "VAR_TZ",
       "type": "constant",
       "label": "Timezone",
-      "value": "America/Los_Angeles",
+      "value": "America/Chicago",
       "description": ""
     }
   ],
diff --git a/dashboards/dashboard-min-mean-max.json b/dashboards/dashboard-min-mean-max.json
index 5af17d6..052f288 100644
--- a/dashboards/dashboard-min-mean-max.json
+++ b/dashboards/dashboard-min-mean-max.json
@@ -27,7 +27,7 @@
       "name": "VAR_TZ",
       "type": "constant",
       "label": "Timezone",
-      "value": "America/Los_Angeles",
+      "value": "America/Chicago",
       "description": ""
     }
   ],
diff --git a/dashboards/dashboard-no-animation.json b/dashboards/dashboard-no-animation.json
index 4bac66e..da5b73c 100644
--- a/dashboards/dashboard-no-animation.json
+++ b/dashboards/dashboard-no-animation.json
@@ -27,7 +27,7 @@
       "name": "VAR_TZ",
       "type": "constant",
       "label": "Timezone",
-      "value": "America/Los_Angeles",
+      "value": "America/Chicago",
       "description": ""
     }
   ],
diff --git a/dashboards/dashboard-simple.json b/dashboards/dashboard-simple.json
index 6fcd549..d61d870 100644
--- a/dashboards/dashboard-simple.json
+++ b/dashboards/dashboard-simple.json
@@ -27,7 +27,7 @@
       "name": "VAR_TZ",
       "type": "constant",
       "label": "Timezone",
-      "value": "America/Los_Angeles",
+      "value": "America/Chicago",
       "description": ""
     }
   ],
diff --git a/dashboards/dashboard.json b/dashboards/dashboard.json
index b09a845..cadbd04 100644
--- a/dashboards/dashboard.json
+++ b/dashboards/dashboard.json
@@ -34,7 +34,7 @@
       "name": "VAR_TZ",
       "type": "constant",
       "label": "Timezone",
-      "value": "America/Los_Angeles",
+      "value": "America/Chicago",
       "description": ""
     }
   ],
diff --git a/influxdb/influxdb.sql b/influxdb/influxdb.sql
index 3beb4b4..99976e7 100644
--- a/influxdb/influxdb.sql
+++ b/influxdb/influxdb.sql
@@ -13,8 +13,8 @@ CREATE RETENTION POLICY grid ON powerwall duration INF replication 1
 CREATE RETENTION POLICY pod ON powerwall duration INF replication 1
 CREATE RETENTION POLICY alerts ON powerwall duration INF replication 1
 CREATE CONTINUOUS QUERY cq_autogen ON powerwall BEGIN SELECT mean(home) AS home, mean(solar) AS solar, mean(from_pw) AS from_pw, mean(to_pw) AS to_pw, mean(from_grid) AS from_grid, mean(to_grid) AS to_grid, last(percentage) AS percentage INTO powerwall.autogen.:MEASUREMENT FROM (SELECT load_instant_power AS home, solar_instant_power AS solar, abs((1+battery_instant_power/abs(battery_instant_power))*battery_instant_power/2) AS from_pw, abs((1-battery_instant_power/abs(battery_instant_power))*battery_instant_power/2) AS to_pw, abs((1+site_instant_power/abs(site_instant_power))*site_instant_power/2) AS from_grid, abs((1-site_instant_power/abs(site_instant_power))*site_instant_power/2) AS to_grid, percentage FROM raw.http) GROUP BY time(1m), month, year fill(linear) END
-CREATE CONTINUOUS QUERY cq_kwh ON powerwall RESAMPLE EVERY 1m BEGIN SELECT integral(home)/1000/3600 AS home, integral(solar)/1000/3600 AS solar, integral(from_pw)/1000/3600 AS from_pw, integral(to_pw)/1000/3600 AS to_pw, integral(from_grid)/1000/3600 AS from_grid, integral(to_grid)/1000/3600 AS to_grid INTO powerwall.kwh.:MEASUREMENT FROM autogen.http GROUP BY time(1h), month, year tz('America/Los_Angeles') END
-CREATE CONTINUOUS QUERY cq_daily ON powerwall RESAMPLE EVERY 1h BEGIN SELECT sum(home) AS home, sum(solar) AS solar, sum(from_pw) AS from_pw, sum(to_pw) AS to_pw, sum(from_grid) AS from_grid, sum(to_grid) AS to_grid INTO powerwall.daily.:MEASUREMENT FROM powerwall.kwh.http GROUP BY time(1d), month, year tz('America/Los_Angeles') END 
+CREATE CONTINUOUS QUERY cq_kwh ON powerwall RESAMPLE EVERY 1m BEGIN SELECT integral(home)/1000/3600 AS home, integral(solar)/1000/3600 AS solar, integral(from_pw)/1000/3600 AS from_pw, integral(to_pw)/1000/3600 AS to_pw, integral(from_grid)/1000/3600 AS from_grid, integral(to_grid)/1000/3600 AS to_grid INTO powerwall.kwh.:MEASUREMENT FROM autogen.http GROUP BY time(1h), month, year tz('America/Chicago') END
+CREATE CONTINUOUS QUERY cq_daily ON powerwall RESAMPLE EVERY 1h BEGIN SELECT sum(home) AS home, sum(solar) AS solar, sum(from_pw) AS from_pw, sum(to_pw) AS to_pw, sum(from_grid) AS from_grid, sum(to_grid) AS to_grid INTO powerwall.daily.:MEASUREMENT FROM powerwall.kwh.http GROUP BY time(1d), month, year tz('America/Chicago') END 
 CREATE CONTINUOUS QUERY cq_monthly ON powerwall RESAMPLE EVERY 1h BEGIN SELECT sum(home) AS home, sum(solar) AS solar, sum(from_pw) AS from_pw, sum(to_pw) AS to_pw, sum(from_grid) AS from_grid, sum(to_grid) AS to_grid INTO powerwall.monthly.:MEASUREMENT FROM powerwall.daily.http GROUP BY time(365d), month, year END
 CREATE CONTINUOUS QUERY cq_pw_temps ON powerwall BEGIN SELECT mean(PW1_temp) AS PW1_temp, mean(PW2_temp) AS PW2_temp, mean(PW3_temp) AS PW3_temp, mean(PW4_temp) AS PW4_temp, mean(PW5_temp) AS PW5_temp, mean(PW6_temp) AS PW6_temp INTO powerwall.pwtemps.:MEASUREMENT FROM (SELECT PW1_temp, PW2_temp, PW3_temp, PW4_temp, PW5_temp, PW6_temp FROM raw.http) GROUP BY time(1m), month, year fill(linear) END
 CREATE CONTINUOUS QUERY cq_pw_tempsb ON powerwall BEGIN SELECT mean(PW7_temp) AS PW7_temp, mean(PW8_temp) AS PW8_temp, mean(PW9_temp) AS PW9_temp, mean(PW10_temp) AS PW10_temp, mean(PW11_temp) AS PW11_temp, mean(PW12_temp) AS PW12_temp INTO powerwall.pwtemps.:MEASUREMENT FROM (SELECT PW7_temp, PW8_temp, PW9_temp, PW10_temp, PW11_temp, PW12_temp FROM raw.http) GROUP BY time(1m), month, year fill(linear) END
diff --git a/telegraf.conf b/telegraf.conf
index b7f786f..a702cab 100644
--- a/telegraf.conf
+++ b/telegraf.conf
@@ -115,12 +115,12 @@
 [[processors.date]]
 	tag_key = "month"
 	date_format = "Jan"
-	timezone = "America/Los_Angeles"
+	timezone = "America/Chicago"
 
 [[processors.date]]
 	tag_key = "year"
 	date_format = "2006"
-	timezone = "America/Los_Angeles"
+	timezone = "America/Chicago"
 
 # Alert Data
 [[inputs.http]]
diff --git a/tz b/tz
index 49244fc..c7aac92 100644
--- a/tz
+++ b/tz
@@ -1 +1 @@
-America/Los_Angeles
+America/Chicago
% 


YesThatAllen avatar May 05 '23 19:05 YesThatAllen

The tz.sh script updates all those files to have your timezone and is run during setup.sh. The updated.sh uses tz.sh to reset to default (yes, America/Los_Angeles) and then does a git pull and runs tz.sh again to set it back to your local timezone as stored in the local (untracked) tz file.

Yes, it is a hack, but the timezone localization required in all those files didn't allow for an external reference.

jasonacox avatar May 06 '23 03:05 jasonacox

Gotcha.. Once we're done with #269, there will be FAR fewer diffs showing, and in my mind, those will be easier to mentally ignore.

Addressing the "hack" is fodder for a later date/issue.

YesThatAllen avatar May 06 '23 13:05 YesThatAllen

Unfortunately, I can't update to the current version due to the following reason:

admin@ds620slim:~/Powerwall-Dashboard$ ./upgrade.sh 
Upgrade Powerwall-Dashboard from 2.9.3 to 2.9.6
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.

Upgrade - Proceed? [y/N] y

Resetting Timezone to Default...

Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...

No local changes to save
remote: Enumerating objects: 79, done.
remote: Counting objects: 100% (79/79), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 79 (delta 45), reused 63 (delta 37), pack-reused 0
Unpacking objects: 100% (79/79), 23.43 KiB | 193.00 KiB/s, done.
From https://github.com/jasonacox/Powerwall-Dashboard
   864bcde..da239f4  main       -> origin/main
 * [new tag]         v2.9.6     -> v2.9.6
 * [new tag]         v2.9.4     -> v2.9.4
 * [new tag]         v2.9.5     -> v2.9.5
Updating 864bcde..da239f4
error: unable to create file influxdb/backups/.keep: Permission denied

Any idea what might be causing this (the last upgrades never failed)?

Some files in ~/Powerwall-Dashboard/influxdb belong to the user root, but as far as I understood, I'm not supposed to run ./upgrade as root:

admin@ds620slim:~/Powerwall-Dashboard/influxdb$ ll
total 60
drwxr-xr-x 1 admin users   358 May  7 14:41 .
drwxr-xr-x 1 admin users   948 May  7 14:41 ..
drwxrwxr-x 1 root  root  14382 Feb 11 15:33 backups
drwxr-xr-x 1 root  root     36 Apr 26  2022 data
-rw-r--r-- 1 admin users  1057 Feb 20 11:18 dropcq.sql
-rw-r--r-- 1 admin users 15280 May  7 14:41 influxdb.sql
-rw-r--r-- 1 admin users 15280 May  7 14:41 influxdb.sql.bak
drwxr-xr-x 1 root  root     14 May  7 02:28 meta
-rw-r--r-- 1 admin users  1089 Feb 11 16:43 run-once-2.8.0.sql
-rw-r--r-- 1 admin users     3 Feb 11 16:46 run-once-2.8.0.sql.done
-rw-r--r-- 1 admin users   508 Feb 20 11:18 run-once-2.8.3.sql
-rw-r--r-- 1 admin users     3 Feb 20 11:18 run-once-2.8.3.sql.done
-rw-r--r-- 1 admin users  3590 May  2 17:52 run-once-2.9.2.sql
-rw-r--r-- 1 admin users     3 May  2 17:52 run-once-2.9.2.sql.done
drwx------ 1 root  root     36 Apr 26  2022 wal

Is it possible that my installation is now corrupted since ./upgrade has already started doing some changes when it failed? I guess that at least the timezone has been changed?

jaydkay avatar May 07 '23 13:05 jaydkay

Any idea what might be causing this (the last upgrades never failed)? Some files in ~/Powerwall-Dashboard/influxdb belong to the user root,

Yep influxdb/backups/ was created by root, so the update script can't create a file in it.

but as far as I understood, I'm not supposed to run ./upgrade as root:

Correct. there are a number of ways we might handle this, but the simplest fix for now is to not attempt to create that .keep file. I've submitted a fix in #272

YesThatAllen avatar May 07 '23 13:05 YesThatAllen

Is it possible that my installation is now corrupted since ./upgrade has already started doing some changes when it failed?

The git operations were called before anything too drastic was attempted on the local setup, so your data should be fine.

(I assume your dashboard is still running and that you can see the grafana interface?)

I guess that at least the timezone has been changed?

The upgrade script should set it back to the timezone you've specified in pypowerwall.env.

YesThatAllen avatar May 07 '23 15:05 YesThatAllen

Merged!

@jaydkay thanks for opening this! This shows a gap in test rigs where I do not have active backup scripts running that change permissions. You should be able to run upgrade.sh again.

Minor correction on timezone - It is set in the tz file when you first ran setup.sh and upgrade will set it back to that after the new release pull.

jasonacox avatar May 07 '23 15:05 jasonacox

@jaydkay I believe I was able to replicate a setup similar to yours (with backup owned by root) and the latest upgrade worked without error.

jasonacox avatar May 07 '23 16:05 jasonacox

Hi @YesThatAllen and @jasonacox, thanks a lot for your quick support! Unfortunately, probably due to the failed upgrade from 2.9.3 to 2.9.6 I now get the following error:

admin@ds620slim:~/Powerwall-Dashboard$ ./upgrade.sh 
Upgrade Powerwall-Dashboard from 2.9.6 to 2.9.7
---------------------------------------------------------------------
This script will attempt to upgrade you to the latest version without
removing existing data. A backup is still recommended.

Upgrade - Proceed? [y/N] y

Resetting Timezone to Default...

Pull influxdb.sql, dashboard.json, telegraf.conf, and other changes...

Saved working directory and index state WIP on main: 864bcde v2.9.3
error: cannot pull with rebase: You have unstaged changes.
error: please commit or stash them.

The upgrade seems to think that I got 2.9.6 installed, but that upgrade failed, so I should still be with 2.9.3. Any idea how to proceed from here?

jaydkay avatar May 07 '23 16:05 jaydkay

try starting with git checkout main

and if git pull --rebase doesn't succeed, review what it says is blocking.

you may need to use git checkout to reset a file or two. Post back if there's issues you can't readily get past, or with other findings?

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

Okay, so:

admin@ds620slim:~/Powerwall-Dashboard$ git checkout main
M       tools/influxdb2/flux/tesla_energy.strings.flux
M       tools/influxdb2/flux/tesla_energy.vitals.flux
Already on 'main'
Your branch is behind 'origin/main' by 15 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

and

admin@ds620slim:~/Powerwall-Dashboard$ git pull --rebase
error: cannot pull with rebase: You have unstaged changes.
error: please commit or stash them.

Sorry, I have no idea how to fix this...

jaydkay avatar May 07 '23 17:05 jaydkay

try git status ?

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

Okay, this gives:

admin@ds620slim:~/Powerwall-Dashboard$ git status
On branch main
Your branch is behind 'origin/main' by 15 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   tools/influxdb2/flux/tesla_energy.strings.flux
        modified:   tools/influxdb2/flux/tesla_energy.vitals.flux

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .editorconfig
        .gitattributes
        backups/backup.sh
        compose.env
        dashboards/archive/
        grafana.env
        grafana/.keep
        grafana/alerting/
        influxdb/backups/
        influxdb/run-once-2.8.0.sql.done
        influxdb/run-once-2.8.3.sql.done
        influxdb/run-once-2.9.2.sql.done
        telegraf.local
        tmp.sh
        tools/solar-only/grafana/.keep
        tools/solar-only/influxdb/.keep
        tools/tesla-history/tesla-history.auth
        tools/tesla-history/tesla-history.conf
        v6.5.1/grafana/.keep
        v6.5.1/influxdb/.keep

no changes added to commit (use "git add" and/or "git commit -a")

jaydkay avatar May 07 '23 17:05 jaydkay

oh.. this is about line endings. you've got the new .gitattributes file which makes your local think the two .flux files have been changed.

rm .gitattributes

and try pulling again?

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

Okay, removed the file and ran git pull --rebase, which now gives:

admin@ds620slim:~/Powerwall-Dashboard$ git pull --rebase
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 9 (delta 5), reused 8 (delta 5), pack-reused 0
Unpacking objects: 100% (9/9), 1.51 KiB | 155.00 KiB/s, done.
From https://github.com/jasonacox/Powerwall-Dashboard
   da239f4..8dd0f81  main       -> origin/main
Updating 864bcde..8dd0f81
error: The following untracked working tree files would be overwritten by merge:
        .editorconfig
        dashboards/archive/dashboard-animation.json
        dashboards/archive/dashboard-grid.json
        dashboards/archive/dashboard-new.json
        dashboards/archive/dashboard-orig.json
        dashboards/archive/dashboard-v2.8.2.json
        dashboards/archive/dashboard-v2.8.3.json
        dashboards/archive/dashboard-v2.9.0.json
        dashboards/archive/dashboard-v2.9.1.json
        grafana/.keep
        tools/solar-only/grafana/.keep
        tools/solar-only/influxdb/.keep
        v6.5.1/grafana/.keep
        v6.5.1/influxdb/.keep
Please move or remove them before you merge.
Aborting

jaydkay avatar May 07 '23 17:05 jaydkay

Gotcha.. so yeah you'd want to remove those as well.

For others playing the home game; the following command, run from the repo's root, would be the thing:

rm rf .gitattributes .editorconfig dashboards/archive/dashboard-animation.json dashboards/archive/dashboard-grid.json dashboards/archive/dashboard-new.json dashboards/archive/dashboard-orig.json dashboards/archive/dashboard-v2.8.2.json dashboards/archive/dashboard-v2.8.3.json dashboards/archive/dashboard-v2.9.0.json dashboards/archive/dashboard-v2.9.1.json grafana/.keep tools/solar-only/grafana/.keep tools/solar-only/influxdb/.keep v6.5.1/grafana/.keep v6.5.1/influxdb/.keep

(leave out the -rf to confirm operations & you'll just have to hit "ok" a bunch)

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

Okay, seems we're getting closer... ;) Deleted all the files and tried again:

admin@ds620slim:~/Powerwall-Dashboard$ git pull --rebase
Updating 864bcde..8dd0f81
Fast-forward
 .editorconfig                                            |  21 +++++++++++++++++++++
 .gitattributes                                           |  25 +++++++++++++++++++++++++
 .github/ISSUE_TEMPLATE/other.md                          |   2 --
 .gitignore                                               |  35 +++++++++++++++++++++++++++++++----
 README.md                                                |  17 +++++++----------
 RELEASE.md                                               |  29 ++++++++++++++++++++++++-----
 VERSION                                                  |   2 +-
 _config.yml                                              |   2 +-
 compose.env.sample                                       |   6 +++---
 dashboard.json                                           |  58 +++++++++++++++++++++++++++++++++++++++++++---------------
 dashboards/README.md                                     |   3 ---
 dashboards/{ => archive}/dashboard-animation.json        |   0
 dashboards/{ => archive}/dashboard-grid.json             |   0
 dashboards/{ => archive}/dashboard-new.json              |   0
 dashboards/{ => archive}/dashboard-orig.json             |   0
 dashboards/{ => archive}/dashboard-v2.8.2.json           |   0
 dashboards/{ => archive}/dashboard-v2.8.3.json           |   0
 dashboards/{ => archive}/dashboard-v2.9.0.json           |   0
 dashboards/{ => archive}/dashboard-v2.9.1.json           |   0
 dashboards/dashboard.json                                |  58 +++++++++++++++++++++++++++++++++++++++++++---------------
 dashboards/powerflow-animation.html                      |   6 +++---
 grafana/.DS_Store                                        | Bin 6148 -> 0 bytes
 grafana/.keep                                            |   1 +
 grafana/README.md                                        |   2 --
 influxdb.conf                                            |   1 -
 influxdb/.DS_Store                                       | Bin 6148 -> 0 bytes
 influxdb/README.md                                       |   2 --
 location.html                                            |   1 -
 powerwall.yml                                            |  11 +++++------
 pypowerwall.sample                                       |   5 -----
 telegraf.local.sample                                    |   4 ++--
 tools/ecowitt-weather-history/ecowitt-weather-history.py |   6 +++---
 tools/influxdb2/flux/tesla_energy.strings.flux           |  52 ++++++++++++++++++++++++++--------------------------
 tools/influxdb2/flux/tesla_energy.vitals.flux            |  66 +++++++++++++++++++++++++++++++++---------------------------------
 tools/solar-only/grafana/.keep                           |   1 +
 tools/solar-only/grafana/README.md                       |   2 --
 tools/solar-only/influxdb/.keep                          |   1 +
 tools/solar-only/influxdb/README.md                      |   2 --
 upgrade.sh                                               |  22 +++++++++++-----------
 v6.5.1/grafana/.DS_Store                                 | Bin 6148 -> 0 bytes
 v6.5.1/grafana/.keep                                     |   1 +
 v6.5.1/grafana/README.md                                 |   2 --
 v6.5.1/influxdb/.DS_Store                                | Bin 6148 -> 0 bytes
 v6.5.1/influxdb/.keep                                    |   1 +
 v6.5.1/influxdb/README.md                                |   2 --
 v6.5.1/pypowerwall.sample                                |   5 -----
 verify.sh                                                |   4 ++--
 47 files changed, 289 insertions(+), 169 deletions(-)
 create mode 100644 .editorconfig
 create mode 100644 .gitattributes
 rename dashboards/{ => archive}/dashboard-animation.json (100%)
 rename dashboards/{ => archive}/dashboard-grid.json (100%)
 rename dashboards/{ => archive}/dashboard-new.json (100%)
 rename dashboards/{ => archive}/dashboard-orig.json (100%)
 rename dashboards/{ => archive}/dashboard-v2.8.2.json (100%)
 rename dashboards/{ => archive}/dashboard-v2.8.3.json (100%)
 rename dashboards/{ => archive}/dashboard-v2.9.0.json (100%)
 rename dashboards/{ => archive}/dashboard-v2.9.1.json (100%)
 delete mode 100644 grafana/.DS_Store
 create mode 100644 grafana/.keep
 delete mode 100644 grafana/README.md
 delete mode 100644 influxdb/.DS_Store
 delete mode 100644 influxdb/README.md
 delete mode 100644 pypowerwall.sample
 create mode 100644 tools/solar-only/grafana/.keep
 delete mode 100644 tools/solar-only/grafana/README.md
 create mode 100644 tools/solar-only/influxdb/.keep
 delete mode 100644 tools/solar-only/influxdb/README.md
 delete mode 100644 v6.5.1/grafana/.DS_Store
 create mode 100644 v6.5.1/grafana/.keep
 delete mode 100644 v6.5.1/grafana/README.md
 delete mode 100644 v6.5.1/influxdb/.DS_Store
 create mode 100644 v6.5.1/influxdb/.keep
 delete mode 100644 v6.5.1/influxdb/README.md
 delete mode 100644 v6.5.1/pypowerwall.sample

I suppose I can now try to normally invoke the ./upgrade.sh script? Thank you!

jaydkay avatar May 07 '23 17:05 jaydkay

Yep, the script had failed on the git pull --rebase. You've done that now so the upgrade script should work as expected.

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

Upgrade will show same version, but you want to continue (technically you should always be able to re-install same version). Also, can you run ./verify.sh to ensure everything is still set up and running correctly?

@YesThatAllen should we add anything to upgrade.sh to accomodate for the repo changes?

jasonacox avatar May 07 '23 17:05 jasonacox

Yes, ./upgrade.sh made an update from 2.9.7 to 2.9.7 and completed without errors. Also, ./verify.shshows that everything is working as expected. Thanks a lot to both of you!

jaydkay avatar May 07 '23 17:05 jaydkay

@YesThatAllen should we add anything to upgrade.sh to accommodate for the repo changes?

The change in 2.9.7 which removed .keep from backups/ should be enough to prevent anyone else running in to this issue.

Other changes to the upgrade & backup scripts are fodder for separate issues.

YesThatAllen avatar May 07 '23 17:05 YesThatAllen

I started putting my thoughts separate issues:

  • potential backup script improvements in https://github.com/jasonacox/Powerwall-Dashboard/issues/273
  • potential upgrade script improvements in https://github.com/jasonacox/Powerwall-Dashboard/issues/274

I imagine we can close this issue once it goes a while without addition related issues like @jaydkay pointed out.

YesThatAllen avatar May 08 '23 00:05 YesThatAllen

Another benefit I hoped to get from this cleanup was to remove the pretty incessant logging of this statement in the grafana docker container:

="ignoring unexpected file while scanning local working directory" filename=/var/lib/grafana/alerting/.DS_Store

I ended up git-pulling and updating my local without the script, then updating docker itself, so the containers restarted. I was still seeing that error.. I don't know enough about how docker populates its containers to know if the .DS_Store file should be expected to get removed.

I was able to remove it manually by using the terminal in the container & running

rm /var/lib/grafana/alerting/.DS_Store

What I actually did

2023-05-12_12-16-38

$ ls -al /var/lib/grafana/alerting/
total 8
drwxr-x--- 4 1000 1000  128 May  5 17:24 .
drwxr-xr-x 8 1000 1000  256 May 12 17:04 ..
-rw-r--r-- 1 1000 1000 6148 May  5 17:24 .DS_Store
drwxr-x--- 3 1000 1000   96 May 12 17:03 1
$ rm /var/lib/grafana/alerting/.DS_Store
$ ls -al /var/lib/grafana/alerting/
total 0
drwxr-x--- 3 1000 1000  96 May 12 17:07 .
drwxr-xr-x 8 1000 1000 256 May 12 17:04 ..
drwxr-x--- 3 1000 1000  96 May 12 17:03 1
$

I assume that no one who's starting this project fresh would see that message. Woot?

YesThatAllen avatar May 12 '23 17:05 YesThatAllen

I noticed we still have a dashboard.json at the top level and a slightly newer copy at dashboards/dashboard.json. Should we cleanup the one at the top level and keep them all in the dashboards folder?

jgleigh avatar Jun 23 '23 11:06 jgleigh

The original purpose of the dashboards folder was to collect user submitted "Alternative Dashboards" as the README.md says. If we move the default dashboard there, we would need to update the setup instructions in setup.sh, upgrade.sh and Readme. I have been reluctant to do that in order to make the setup easy (keeping what you need in the root folder) but the duplicate file is probably a good reason to reconsider.

jasonacox avatar Jun 24 '23 06:06 jasonacox

I'm fine leaving it in the root folder, but then we should remove the duplicate in the dashboards folder. The one in dashboards had some slight tweaks to two of the queries.

jgleigh avatar Jun 24 '23 14:06 jgleigh

I think that proves that we should clean it up and just use the one in the dashboards folder. It is logical and shouldn't be difficult for the community to figure out. I updated the instructions including the setup.sh and upgrade.sh references to the new location. Thanks for raise this @jgleigh ! 🙏

jasonacox avatar Jun 25 '23 22:06 jasonacox

The tz.sh script updates all those files to have your timezone and is run during setup.sh. The updated.sh uses tz.sh to reset to default (yes, America/Los_Angeles) and then does a git pull and runs tz.sh again to set it back to your local timezone as stored in the local (untracked) tz file.

Yes, it is a hack, but the timezone localization required in all those files didn't allow for an external reference.

What about to use template files and create the final ones from them? The template files would have $TZ as reference, and then envsubst command could be used to change it and write the final files.

rcasta74 avatar Jul 26 '23 20:07 rcasta74