Add history level persisting process instance variable without full activity history

[GeorgiPetkov]

Is your feature request related to a problem? Please describe.

I've been using FULL history level so far. There's a lot of redundant data that I don't need (and the tables get huge - tens of GBs) - mainly activity history (act_hi_actinst) and variable changes (act_hi_detail). I can easily get rid of the second one by using history level ACTIVITY.

Unfortunately, the activity's history is coupled with the variable's history. I really need the variables. I'm surprised that the INSTANCE history level is not including them. I really can't see what's the use of persisting instances without the variables.

Describe the solution you'd like

Ideally from my PoV the INSTANCE history level should also persist variables. If this is not a bug and it really makes sense for some users, then another level achieving the same thing works as well.

Describe alternatives you've considered

I'm considering background jobs clearing the act_hi_actinst table periodically.

Additional context

What further makes it worse is that:

• I'm having long-running processes with polling which further generates more activity history (by looping on the diagram).
• The variable updates are stored even when not changing anything (i.e. deleting not existing variable or setting the same value again).

[tijsrademakers]

Flowable has more history levels than FULL and INSTANCE, You can use AUDIT or ACTIVITY level like you are saying, and this will still store the variable data in the history. What it doesn't do is store all variable changes in the history, that is only done in the FULL level, is that what you need?

The INSTANCE level history is a very minimal history level and therefore doesn't include the variable data.

You can delete historic data using the HistoryService bulkDeleteHistoricProcessInstances for example.

[GeorgiPetkov]

I want to persist instances and variables, but not activity history and variable changes.

[wberges]

@GeorgiPetkov be careful because it's not only the HIstory tables which grow, there's also ACT_RU_ACTINST which is the runtime history of all activities and grows a lot when you loop in your workflows (only deleted when the workflow ends). See https://forum.flowable.org/t/where-is-act-ru-actinst-used-and-can-i-delete-data-from-it/6986/1

[GeorgiPetkov]

@GeorgiPetkov be careful because it's not only the HIstory tables which grow, there's also ACT_RU_ACTINST which is the runtime history of all activities and grows a lot when you loop in your workflows (only deleted when the workflow ends). See https://forum.flowable.org/t/where-is-act-ru-actinst-used-and-can-i-delete-data-from-it/6986/1

@wberges Thanks for the heads up. I've noticed that, but it's fine for my use cases. The process instances won't last over several days and on average there are no more than 10 running in parallel, so it's perfectly manageable. I still support your request, it would be even more efficient for systems with higher load.