Transaction heuristic state should be reported in better way

[ochaloup]

When transaction scale down process reviews the state of transactions in the pods it checks if there are no unfinished prepared transactions. If there are found some then it does not permit the pod being terminated and the pod is moved to state SCALING_DOWN_RECOVERY_DIRTY. It's left there until all transactions are safely recovered.

There are cases when transaction is never recovered in automated manner. It's time when transaction is in heuristic state. Such transaction needs to be handled manually by administrator and he is expected to connect to the app server with jboss-cli.sh and verify the transaction state.

The operator should report better that there is a transaction in heuristic state.

• a new state should be created for the operator to report that the txn log is dirty and heuristics
• it would be good to emit event - in the best the event would be emitted for pod and for the operator as well - which announces that the pod contains heuristic transactions which can't be managed automatically
• a counter of the recovery attempts should be held for the pod (maybe by adding a new status value)

[ochaloup]

@jmesnil submitting this as an issue that I would like to work on but just after the WFLY18/CD18 release.

[ochaloup]

In connection to https://github.com/wildfly/wildfly-operator/issues/109

[jmfinelli]

I am working on this issue ATM. IMO, SCALING_DOWN_RECOVERY_DIRTY does not make much sense. Also, in the moment we create a new state for heuristic transactions (e.g. SCALING_DOWN_HEURISTIC), we introduce even more confusion on the differences between DIRTY and HEURISTIC. My suggestion is to rename SCALING_DOWN_RECOVERY_DIRTY to SCALING_DOWN_RECOVERY_UNCOMPLETED or SCALING_DOWN_RECOVERY_PROCESSING.

[ochaloup]

To me this suggestion sounds reasonable.