scio icon indicating copy to clipboard operation
scio copied to clipboard

Published 20 hours ago • verified publisher icon spotify

Double check / refine post execution hooks

Open nevillelyh opened this issue 3 years ago • 1 comments

Something to execute after sc.run() with ScioContext & PipelineResult, for integration hooks i.e. submitting counters to a dashboard, lineage, or updating Bigtable cluster.

nevillelyh avatar Nov 05 '20 15:11 nevillelyh

IIRC we removed those hooks in Scio 0.8.x since all the Future were removed. Now you can just do:

// ...
sc.waitUntilDone()
executePostProcessing()

Are you suggesting we reintroduce hooks ?

jto avatar Nov 12 '20 14:11 jto