write results to file

[peshay]

I like to get the test_results written to a file, so a build-server can later pick it up. When I used your role from ansible-galaxy, I could add in your main.yml

- name: write report
  local_action: copy content="{{ goss_output.stdout }}" dest="{{ goss_report_file }}"
  become: false
  when: goss_report_file is defined

I just wanted to make this a pull requests, but see that you changed a lot here! ;) Still I would file a pull request for that feature, but maybe you can already tell me what you think about this idea?

[naftulikay]

@peshay I'm stoked to have users! :raised_hands: :sun_with_face:

I was actually considering doing something similar to this, but I wasn't sure how to make it ergonomic and flexible enough for users. The current use-case that is supported is:

• Set degoss_clean_on_failure to false, and when failures occur, grab {{ degoss_tmp_root }}/result.json and examine it to understand the failures.

The problem with this current approach is that it's not at all easy to get degoss_tmp_root on failure. On success, the next play should see the degoss_tmp_root fact, but if it fails, currently it just aborts.

I guess I'm just asking what the exact use case is here. Do you want the role to never fail, even on test failure? Would adding degoss_result_file and copying the result there, regardless of pass/fail, satisfy your use-case?

[peshay]

My use case is to have my role development checked by a build server. So it's totally fine, that if something fails in the goss checks, that the role fails and stops further execution and that the build shows as failed. But in any case I want the report output in a file, so the jenkins tap plugin can grab the report and I could see which checks failed and so on.

[naftulikay]

@peshay thanks for getting back.

So the new refactor only and exclusively uses the JSON output so as to make handling failure cases easier. There is an outstanding bug (aelsabbahy/goss#317) that Goss returns 0 on failure at times, so the only reliable way for us to detect failure is to use the JSON output and try to deserialize the output into JSON. If deserialization fails, we know that Goss encountered a critical error, otherwise we report success and failure.

So presently, the only output format supported going forward will be JSON. It should be possible to translate the JSON into Jenkins tap format, but yeah, I don't know if we can still support different output formats :(

If I added degoss_result_file to store the JSON in, would this still be helpful?

I had considered executing with JSON output and then subsequently again with the desired other output format, but this would require all tests to be idempotent, which is really error-prone.

[naftulikay]

Oh, also, there may be a workaround to make it easy to catch events. See callback_plugins/goss_format.py here in the repository for an example of how to do this. Catch any result from a degoss task and you'll be able to store or otherwise work with the JSON.