Hooks validation tests cannot pass

[nixpig]

Running the validation/hooks/hooks.t tests should pass, but they always fail.

For example, using runc:

RUNTIME=runc ./validation/hooks/hooks.t
TAP version 13
  ---
  {
    "error": "Hooks MUST be called in the listed order\nRefer to: https://github.com/opencontainers/runtime-spec/blob/v1.1.0/config.md#posix-platform-hooks\nThe runtime MUST run the user-specified program, as specified by `process`\nRefer to: https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#lifecycle"
  }
  ...
1..0

Seems to be two issues here:

1. The bundle directory (where the output file is stored) is deleted (in RuntimeLifecycleValidate) before the output file is read for comparison: https://github.com/opencontainers/runtime-tools/blob/f7e3563b0271e5cd52d5c915684ea11ef2779572/validation/hooks/hooks.go#L77-L78 Because of this:
   1. The content of outputData is always empty.
   2. We don't see the 'file not found' error from os.ReadFile because it's ignored.
2. Even if outputData was populated per the specified hooks, the comparison of its content will always fail: https://github.com/opencontainers/runtime-tools/blob/f7e3563b0271e5cd52d5c915684ea11ef2779572/validation/hooks/hooks.go#L79
   1. Actual content: pre-start1 called\npre-start2 called\npost-start1 called\npost-start2 called\npost-stop1 called\npost-stop2 called\n
   2. Asserted content: pre-start1 called\npre-start2 called\npost-start1\npost-start2\npost-stop1\npost-stop2\n
   3. Note the lack of called postfix for post-startx and post-stopx hooks.

[nixpig]

PR with proposed fix: #782

[ghost]

Running the validation/hooks/hooks.t tests should pass, but they always fail.

For example, using runc:

RUNTIME=runc ./validation/hooks/hooks.t
TAP version 13
  ---
  {
    "error": "Hooks MUST be called in the listed order\nRefer to: https://github.com/opencontainers/runtime-spec/blob/v1.1.0/config.md#posix-platform-hooks\nThe runtime MUST run the user-specified program, as specified by `process`\nRefer to: https://github.com/opencontainers/runtime-spec/blob/v1.1.0/runtime.md#lifecycle"
  }
  ...
1..0

Seems to be two issues here:

1. The bundle directory (where the output file is stored) is deleted (in RuntimeLifecycleValidate) before the output file is read for comparison: https://github.com/opencontainers/runtime-tools/blob/f7e3563b0271e5cd52d5c915684ea11ef2779572/validation/hooks/hooks.go#L77-L78 Because of this:

   1. The content of outputData is always empty.
   2. We don't see the 'file not found' error from os.ReadFile because it's ignored.

2. Even if outputData was populated per the specified hooks, the comparison of its content will always fail: https://github.com/opencontainers/runtime-tools/blob/f7e3563b0271e5cd52d5c915684ea11ef2779572/validation/hooks/hooks.go#L79

   1. Actual content: pre-start1 called\npre-start2 called\npost-start1 called\npost-start2 called\npost-stop1 called\npost-stop2 called\n
   2. Asserted content: pre-start1 called\npre-start2 called\npost-start1\npost-start2\npost-stop1\npost-stop2\n
   3. Note the lack of called postfix for post-startx and post-stopx hooks.

[ghost]

https://github.com/opencontainers/runtime-tools/issues/781#issue-2697274422