Performance of make_basic_network

[mtanneau]

The make_basic_network function is slow for large networks. The timings below were obtained with @time make_basic_network(data) where data was loaded with the PGLib package.

Case	#buses	time (s)	memory (MiB)
1354_pegase	1354	0.21	28.5
2869_pegase	2869	0.62	66.8
9241_pegase	9241	2.50	208.4
13659_pegase	13659	5.72	303.1
30000_goc	30000	15.81	467.8

I profiled it on the 9241_pegase case:

which shows that, besides an initial deepcopy which we can't do much about, a substantial amount of time is spent identifying connected components. Note that the very high stacktrace with _cc_dfs corresponds to recursive calls to a DFS implementation here.

PR coming

[mtanneau]

@ccoffrin I have opened two PRs to address the initial deepcopy (#915) and the identification of connected components (#914).

The 3rd room for performance improvement is within _renumber_components, which contains a lot of deepcopys itself. Namely, all individual components are deepcopied before being re-numbered.

I believe this is not necessary, since we deepcopy the initial data dictionary, and everything afterwards modifies the data dict in-place. As far as I can see, _renumber_components is only called inside make_basic_network, so eliminating the deepcopy below would save further time and memory https://github.com/lanl-ansi/PowerModels.jl/blob/8e346b48c187b2a53907dc031603482f2e6a2017/src/core/data_basic.jl#L156-L160

[ccoffrin]

Has this been addressed by https://github.com/lanl-ansi/PowerModels.jl/pull/919?

[mtanneau]

Yes!

For completeness, I re-ran the timings and profiles on master; results are below.

---

New timing table on master for make_basic_network:

Case	#buses	time (s)	memory (MiB)
1354_pegase	1354	0.13	20.1
2869_pegase	2869	0.29	45.7
9241_pegase	9241	1.09	137.1
13659_pegase	13659	1.47	215.5
30000_goc	30000	2.53	320.1

and corresponding profile on 9k Pegase system

---

Same, but for the in-place make_basic_network!:

Case	#buses	time (s)	memory (MiB)
1354_pegase	1354	0.05	11.0
2869_pegase	2869	0.13	27.2
9241_pegase	9241	0.47	74.7
13659_pegase	13659	0.70	130.0
30000_goc	30000	1.27	181.6

and corresponding profile on 9k Pegase system