Exclude fields in Hoverfly Diff Mode

[cvadityaflipkart]

Is your feature request related to a problem? Please describe. The diff mode in Hoverfly asserts on all the fields. It would be quite useful if there is a way to exclude fields during the assert.

Describe the solution you'd like A solution on the lines of matching strategies present in simulation mode could be used when in diff mode. Also support for JSONPath and XPath should be used as matchers in the exclusion.

[tommysitu]

You can ignore response headers when diffing:

Usage:
  hoverctl mode [capture|diff|simulate|spy|modify|synthesize (optional)] [flags]

Flags:
      --all-headers                          Record all request headers (for capture mode) or ignore all response headers (for diff mode)
      --headers Content-Type,Authorization   A comma separated list of request headers to record (for capture mode) or response headers to ignore (for diff mode) Content-Type,Authorization

But it's very limited. What you suggested is good. We did something similar to filter Journal: https://docs.hoverfly.io/en/latest/pages/reference/api/api.html#post-api-v2-journal

[kapishmalik]

@tommysitu I can work on this issue.

[kapishmalik]

@tommysitu fyi I am busy with some work.. I may take some time to complete this.

[kapishmalik]

@tommysitu I see a few problems with this.

• In diff mode, if the response is other than JSON then it compares the complete body and tells difference.
• Secondly, we are not storing the actual response returned during diff mode and comparison happened. We are just storing the difference.
• Lastly, in case of JSON, we are comparing each field and returning DiffEntry per field/key in that JSON.

Also, I wanted to know why we want to do this enhancement if we are returning the difference per field/key in JSON. Users can basically ignore that in the response. It will be weird for users to tell us which field to ignore and we will just not send that field difference back. Also, if we do this, we need to store actual responses returned at the time of diff mode. Also, I was thinking if we do this then better to take this as configuration for diff mode instead of introducing POST requests for searching and filtering. We are already taking headers.

[kapishmalik]

@tommysitu let me know how to proceed here?

[tommysitu]

I think the purpose of diff mode is to detect API response changes (especially breaking ones but it's difficult to derive from the captured data), if that happens, the captured simulation should be invalidated and one can re-capture to ensure the simulation is up-to-date.

Since some response fields in JSON have volatile data, like id or timestamps, if we don't exclude them from comparing, they would definitely trip the diffing and cause false positives. I think currently diff mode generates a report for this. The original suggestion was that we should provide a way to exclude these JSON fields given the JSONPath expressions, or in XML cases, using XPath expressions. Do you think this is easy to do?

However it could work for a small number of diffs, but for a large response body, manually applying exclusions might not be scalable. The best way is probably to run diff mode right after capture mode, as we know that the API is unlikely to have a breaking change immediately, the diffings that are found would be most certainly noises, and can be excluded for future diffing. If we have an automated way to generate exclusion rules from initial diffings like this, then the feature would be more attractive.

[kapishmalik]

@tommysitu I am thinking to add support for json only to start with. I will be taking jsonpath from the user and converting jsonpath( for an example $.id) into relative paths ( example /id/). Also, instead of writing jsondiff our own, I am planning to use jsondiff library https://github.com/wI2L/jsondiff to calculate the patch and add entries in diffEntry only for replace and remove operations to keep response same as we were showing earlier. We ll be removing those from responses which user asked to exclude. Let me know if this works?

[tommysitu]

@kapishmalik just try to understand here, you are going to use jsondiff to diff the captured response and the new response? It will find the patch and we can exclude those ones user have provided to get the final diffs. Is that right? It sounds like a plan

[kapishmalik]

@cvadityaflipkart we have added support. https://docs.hoverfly.io/en/latest/pages/reference/api/api.html#post-api-v2-diff