teraslice icon indicating copy to clipboard operation
teraslice copied to clipboard

Stateful jobs should be triggered automatically by the use of a stateful asset

Open godber opened this issue 1 year ago • 0 comments

We have this notion of "Stateful Workers" as described here:

https://terascope.github.io/teraslice/docs/configuration/clustering-k8s/#stateful-workers

In Teraslice jobs you can set the "stateful": true, property and the workers will be slightly different (see docs). I've realized that this is actually a property of the asset in use, and should probably be set automatically simply by using a "stateful" asset.

In general this is the stateful processor being used:

https://github.com/terascope/elasticsearch-assets/tree/master/asset/src/elasticsearch_state_storage

This probably needs some internal API added to Teraslice that the processor will use to indicate that it is stateful. Perhaps a property on the processor that Teraslice uses.

godber avatar Oct 01 '24 20:10 godber