Builtin Execution Hooks

[b-butler]

Feature description

#508 Has been merged which provides the base implementation for hooks. However there are currently no provided builtin hooks for use by the end user. The potential hooks that were implemented in #189 are a good starting point to discuss what we should provide batteries for so to speak. Discussion on the appropriate hooks (including ones not messaged) are welcome.

• LogOperation - Log basic information about the execution of operations to a log file.
• TrackOperations - Keep detailed metadata records about the state of the project root directory and the operation, including directives, to a log file, optionally in conjunction with git.
• SnapshotProject - Create a snapshot of the project root directory to keep track of the code used for the execution of an operation.
• TrackWorkspaceWithGit - The workspace is treated as a git repository and automatically committed to before and after the execution of an operation. The can be done either on a per-job basis or workspace global.

Proposed solution

Whoever implemented the builtin hooks would just need to work with the existing hook infrastructure and add the code to perform the specified actions on the correct triggers. These functors/functions can then be passed to Project.operation_hooks.on_... and project level hooks as desired.

[klywang]

Whoever implemented the builtin hooks would just need to work with the existing hook infrastructure and add the code to perform the specified actions on the correct triggers. These functors/functions can then be passed to Project.operation_hooks.on_... and project level hooks as desired.

So will builtin hooks exclusively be used in the following way (A)?

from flow.hooks import LogOperations

@Project.operation
@Project.operation_hooks.on_start(LogOperations("foo.log").log_operations)
def foo(job):
    """ Do something """

Do we want any operation-level built in hooks like this (B)?

@Project.operation
@Project.operation_hooks.on_start.log("foo.log")
def foo(job):
    """ Do something """"

(A) is pretty straightforward to implement. (B) would require rewriting _HooksRegsiter, but I think (B) is a little more intuitive. Not sure if people think it's worth the extra work though.

[bdice]

I strongly favor (A). We want to treat hooks like a "plugin system" for which we provide a handful of nice plugins, rather than treating each of the hooks provided by signac-flow as a built-in feature. The hooks feature is meant to be an entry point or abstraction for user-provided code (callbacks), and we happen to supply a handful of popularly requested examples.

It is important to recognize that the point of leverage here is the injection of the callbacks into the execution loop. We want to be able to independently write and test the two components: the hooks system itself, and the functionality of a specific hook. Blending these by adding a feature like on_start.log makes it more difficult to separate the registration and execution of a hook callback from the logic of the hook function itself.

If a user wants to write syntactic sugar around hooks, to condense it, they can create a custom decorator and apply it to several operations.