nbn-upgrade-map Track technology and upgrade tallies over time. Expose on website.

There's already a function to generate this - get_tech_and_upgrade_breakdown() - but it just emits output to the console. It does take several minutes to run though, so shouldn't be included in every cycle.

All tech breakdown: 14652833
Counter({'FTTN': 4611720,
         'HFC': 2886497,
         'FTTP': 2793945,
         'FTTC': 1696950,
         'WIRELESS': 969169,
         'SATELLITE': 654873,
         'FTTB': 650671,
         'NULL': 389008})
All upgrade breakdown (excluding tech=FTTP): 11858888
Counter({'NULL_NA': 8339191,
         'FTTP_SA': 2030854,
         'UNKNOWN': 1351746,
         'FTTP_NA': 130383,
         'FW_NA': 4215,
         'FW_CT': 1767,
         'FTTN_CT': 552,
         'SAT_CT': 151,
         'FTTC_NA': 16,
         'FTTN_NA': 12,
         'FTTC_CT': 1})

Sep 18 '23 23:09 lyricnz

Yeah this would be a very interesting statistic to watch over time, we could also partially backfill using the git history.

Sep 18 '23 23:09 LukePrior

I'm experimenting with running this code on old checkouts of /results to generate historical data.

Roughly:

    while co_date > datetime(2021, 7, 1):
        co_date -= timedelta(days=7)
        cmd = f"git checkout `git rev-list -n 1 --before=\"{co_date.strftime('%Y-%m-%d %H:%M')}\" main`"
        get_tech_and_upgrade_breakdown()
        os.rename("results/breakdown.json", f"results/breakdown-{co_date.strftime('%Y-%m-%d')}.json")

Sep 19 '23 00:09 lyricnz

That's being somewhat stymied by the API changes a couple of months back (combined-suburbs.json etc). Will change to a pure geojson verison...

Sep 19 '23 03:09 lyricnz

Running against git history - takes about 2 minutes per cycle (currently running every 7 days, starting now moving backwards)

Sep 19 '23 04:09 lyricnz

Pushed PR and output https://github.com/LukePrior/nbn-upgrade-map/pull/266/files#diff-bbd7c3dbacf5aed3a6ba4d4847412e11a7b8098f831e6e660560a2823365d55f

Sep 19 '23 04:09 lyricnz

FWIW project got to 100% address coverage on 31 July

Sep 19 '23 04:09 lyricnz

Oh woah this looks great, might be a good idea to have an option to only include data from July 31st onwards.

Also maybe fixing the date so it will always be one week apart and new data can be added by just generating the missing weeks and appending to the file.

Sep 19 '23 05:09 LukePrior

Changed the processing to go old-to-new, starting on 23/5/23, and using 00:00 timestamp.

Sep 19 '23 05:09 lyricnz

In terms of getting it onto the site, I might just quickly make a new page /stats or something and generate a few charts using Chart.js

Sep 19 '23 06:09 LukePrior

Mockup I made with ChatGPT: https://lukeprior.github.io/nbn-upgrade-map/stats

Sep 20 '23 05:09 LukePrior

I'll fix the code to make it update the same file, so it's rerunnable, without lots of little files. Give me 30 mins

Sep 20 '23 23:09 lyricnz

The breakdown.json file seems to have some issues at the start with dates and values.

Sep 21 '23 01:09 LukePrior

Yup, I think I broke it while reformatting/merging it. I'll just nuke it and rebuild it. Edit: Done

Sep 21 '23 01:09 lyricnz

Merged, ~will update site~ then need to make GHA for automation each week.

Sep 21 '23 02:09 LukePrior

I don't know if the git commands (in particular the checkout) will work within GHA. We may need a different version of the code which just appends progress.json with the current checkout values.

Sep 21 '23 02:09 lyricnz

Maybe a table on the stats page, for easier reading without lots of mouseover?

Sep 21 '23 03:09 lyricnz