Nik Cubrilovic

https://github.com/opennem/opennem/wiki/Network-Data

@gmmspi I was going to setup an instance of docusaurus on `docs.opennem.org.au` https://docusaurus.io/docs merge the wiki and the current dev documentation from `opennem-py/docs` there

https://github.com/opennem/opennem/wiki/Networks this would make more sense as a graphic in terms of for what time periods where the data is coming from

We had to throttle and restrict data, because as an open API with no auth we were being hit too hard by crawlers and users - and didn't have the...

``` run_flow_update_for_interval(datetime.fromisoformat("2023-03-07T00:00:00+10:00"), network=NetworkNEM, number_of_intervals=1) ``` on dev flamegraph - it's all in io wait from pandas for the query https://gist.githubusercontent.com/nc9/310d0f8fe024c4b6db9dba44db285300/raw/1bd3a16e1741b2abab085d142c1a1b5aef0fc804/run_test.py-2023-03-07T12:37:06+11:00.svg

This query: https://github.com/opennem/opennem/blob/master/opennem/aggregates/network_flows.py#L71 ```sql select t.trading_interval, t.network_region, sum(t.power) as generated, sum(t.market_value) as market_value, sum(t.emissions) as emissions from ( select fs.trading_interval as trading_interval, f.network_region as network_region, sum(fs.generated) as power, coalesce(sum(fs.generated) *...

```csv "trading_interval","network_region","generated","market_value","emissions" "2023-03-06 13:55:00+00","NSW1",6421.208179,511962.92811167,4537.1484928540295 "2023-03-06 13:55:00+00","QLD1",7191.572573,501468.35551529,6243.178760527726273404999 "2023-03-06 13:55:00+00","SA1",1814.54734,-5625.096754,88.7169226214130041 "2023-03-06 13:55:00+00","TAS1",578.898799,6425.7766689,0 "2023-03-06 13:55:00+00","VIC1",5684.846761,113.69693522,3421.8557410616931 ```

json writing flame-graph https://gist.githubusercontent.com/nc9/2122d15b3445f2a030baca002b0a3ebd/raw/bdd1f607cc76087234e651b1eec3eac4e233bc0f/run_test.py-2023-03-07T13:57:50+11:00.svg 23 seconds total time

3 seconds now, with ~600ms of network latency https://gist.githubusercontent.com/nc9/9f3a259067d3c5af67438a0018a4032d/raw/b4275b5bbb2c90105582b3ec33d539aca644f7ab/run_test.py-2023-03-08T08:14:45+11:00.svg https://explain.dalibo.com/plan/461c4fa629gd4852

```py def calculate_flow_for_interval(df_energy_and_emissions: pd.DataFrame, df_interconnector: pd.DataFrame) -> pd.DataFrame: """Calculate the flow for a given interval df_energy_and_emissions: generation and emissions data for each network region - energy (MWh) - emissions (tCO2)...