Add reinitialize_charm to harness

[sed-i]

This PR adds a convenience method to harness for simulating a charm upgrade (example).

Manually emitting upgrade_charm could probably be enough to cover most of the need, but having all the correct events emitted in the correct order makes it easier to choose unit tests over integration tests where possible.

[rwcarlsen]

This definitely looks useful. However, particularly after discussions at our recent roadmap meeting, I think this sort of thing probably needs to start as a library. Maybe we need to come up with an ops framework "boost" library that contains high-power experimental libs (that selectively get merged into the framework). The issue here is that there are all sorts of event sequences a charm should be checking. It especially gets complex when you account for things like interleaving processes - e.g. upgrade during scaling op, etc. Or even simply - losing workload connectivity at various points in a sequence. If there are several things charms want to test, then which ones do we want to promote as "special" for testing with the harness? In general I think just adding more special "Harness.simulate_sequence_[foo]" methods will not necessarily be the best/scalable approach. Creating tools/helpers to assist and guide charm testing along these lines is something we will want to improve and work on from the framework side.

[PietroPasotti]

I can agree with that as well, but I think that the 'reinitialize_charm' method would be a generic utility worthy of making it directly into the harness. I've already written three utility functions for that same functionality in as many projects.

[rwcarlsen]

When I've needed to "reset" things for testing, I just create a new Harness instance. Are there use cases y'all have run into where that doesn't work well?

[sed-i]

In general I think just adding more special "Harness.simulate_sequence_[foo]" methods will not necessarily be the best/scalable approach.

I agree in general but in my mind upgrade is in the same precious category as begin_with_initial_hooks. To be more specific: operator-template could have a generic unit test baked in that all charms must pass all the time: begin_with_initial_hooks followed by upgrade must pass (not raise).

[rwcarlsen]

begin_with_initial_hooks' existence itself is considered controversial by some ;-) I'm more open to a generic reinit/reset a charm functionality. I know G and M have expressed a general concern for expanding the corpus of testing/event sequences in begin_with_initial_hooks fashion.

[sed-i]

begin_with_initial_hooks' existence itself is considered controversial by some

I get that, but when a test can be a unit test or an integration test, I'd rather it be a unit tests, and that's where "generic event sequencing" really shines.

[sed-i]

Side note: a hypothesis extension for harness could be very interesting for test scalability and coverage. Silly example:

from ops.testing.strategy import happy_path
from hypothesis import given

@given(sequence=happy_path(...))
def test_startup(self, sequence):
    harness = Harness(MyCharm, sequence=sequence)

[PietroPasotti]

Side note: a hypothesis extension for harness could be very interesting for test scalability and coverage. Silly example:

from ops.testing.strategy import happy_path
from hypothesis import given

@given(sequence=happy_path(...))
def test_startup(self, sequence):
    harness = Harness(MyCharm, sequence=sequence)

Regarding the hypothesis extension: see https://github.com/PietroPasotti/charm-events/blob/main/simulator.py

[sed-i]

Reduced scope to reinitialize only. This now feels a bit confusing:

• What does reinitialize mean outside the context of an upgrade?
• Should statuses, relations, relation data be retained or dropped across a re-init?

[rwcarlsen]

Definitely good questions. From a high-level conceptual view, I think reinit needs to basically enable/facilitate writing event-sequence tests without requiring test authors to use any internal/private ops/harness interfaces.

[rwcarlsen]

I just realized that this PR seems to be somewhat related to this issue: https://github.com/canonical/operator/issues/736

[sed-i]

Abandoned. Closing.