Speed up ledger tests

[meddle0x53]

Motivation

Part of the work required for #2996

Testing some ideas with shared genesis block and some heavy calls removed or reworked.

Test Plan

Monitor the CI for speedups, but even before that try locally to see if there is a visible speed up.