[RFC] JSON result comparison with regular expression support

[Toflar]

Hey guys,

I thought I'll open an issue for the general concept before I work on a PR for this. I very often find myself using something like

And the JSON node "uuid" should not be null
And the JSON node "street" should be null
And the JSON nodes should be equal to:
   | @context  | /contexts/Organisation     |
   | @type     | Organisation               |
   | title     | Acme |

because I cannot use "And the JSON should be equal to" as I get a new UUID every time the test runs again. I find it very complicated to write these tables and copy all the values just because I cannot compare the whole document. Also, because the table uses strings to compare, I have to use a different expression to compare null values (And the JSON node "street" should be null), as if I'd add null to the table it would compare it to the string value "null" instead.

That's why I came up with the idea of having a special regular expression comparison within the document. Here's what I'm thinking of:

    And the JSON should be equal to considering regular expressions:
    """
    {
      "@context": "\/contexts\/Organisation",
      "@id": "!regex:@\\/organisations\\/[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}@",
      "@type": "Organisation",
      "title": "Acme",
      "street": null
    }
    """

So there's a special !regex: control string that allows you to validate a value for a custom regular expression within ". Internally it would evaluate the regular expression and fail if that regular expression does not match. If it does match, the value gets replaced with the value of the original JSON so after testing/replacing all !regex: instructions, we can normally compare the two JSON documents. That would simplify a lot of my tests and make them better because with the current version, what happens if I add a new property? The tests would all be green. In the new variant it would fail because I'm not expecting to get a new property 😄

Do you like that concept? If so, any preference regarding !regex: or would that be fine? And if I'd work on a PR for that, how fast could that be released in a new version? (asking because I'd need it for a project).

[sanpii]

because I cannot use "And the JSON should be equal to" as I get a new UUID every time the test runs again.

For this case, I prefer using json schema.

Do you like that concept?

I think mixing json and regex is unreadable but your idea with !regex: prefix improve this (the regex is located to a small part of json).

And if I'd work on a PR for that, how fast could that be released in a new version? (asking because I'd need it for a project).

The next version is ready, I publish it when a sufficient number of person ask me to publish it. I can wait your PR and publish the 3.0 version after.

[Toflar]

For this case, I prefer using json schema.

Yeah but you cannot check the values, only the keys.

I think mixing json and regex is unreadable but your idea with !regex: prefix improve this (the regex is located to a small part of json).

Yeah it‘s just another tool you can use. You should try to choose wisely and find the best option :)

I‘ll work on a PR then :)

[sanpii]

Yeah but you cannot check the values, only the keys.

Of course you can: http://json-schema.org/example2.html#the-diskuuid-storage-type

[Toflar]

I think you don't get what I'm trying to test. The UUID regex test here is not the main advantage. I could've implemented a keyword like

"uuid": "---placeholder---",

to accept whatever value in there. The schema validation would allow allow to test for a valid UUID, yes, but I cannot test all the rest of the response to check for the values. Maybe I'll try to illustrate it again. Given I have the following response:

{
  "@context": "\/contexts\/Organisation",
  "@id": "\/organisations\/39C147DE-C84B-48BF-9D38-4F66202D1080",
  "@type": "Organisation",
  "title": "Acme",
  "street": null
}

I want to do the following assertions:

• Test the response contains valid JSON
• Test the content of @context, @id, @type, title and street whereas I do not care about the real value of @id, it just has to contain a valid uuid with the \/organisations\/ path.
• Test the response only contains these 5 keys and not more.

To do that, as of today, I need to do this:

Then the response should be in JSON
And the JSON should be valid according to this schema:
"""
{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "required":true,
    "properties": {
        "@context": {
            "type": "string",
            "required":true
        },
        "@id": {
            "type": "string",
            "pattern": "^\\/organisations\\/[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
            "required":true
        },
        "@type": {
            "type": "string",
            "required":true
        },
        "title": {
            "type": "string",
            "required":true
        },
        "street": {
            {"type": ["string", "null"]},
            "required":true
        }
    }
}
"""
And the JSON nodes should be equal to:
  | @context  | \/contexts\/Organisation |
  | @type     | Organisation             |
  | title     | Acme                     |
And the JSON node "street" should be null

All of this is needed to do all my desired assertions. Now compare to this:

And the JSON should be equal to considering regular expressions:
"""
{
  "@context": "\/contexts\/Organisation",
  "@id": "!regex:@\\/organisations\\/[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}@",
  "@type": "Organisation",
  "title": "Acme",
  "street": null
}
"""

[guilliamxavier]

@Toflar For your desired assertions, you actually can do this (note enum and additionalProperties):

And the JSON should be valid according to this schema:
"""
{
    "type": "object",
    "$schema": "http://json-schema.org/draft-03/schema",
    "required":true,
    "properties": {
        "@context": {
            "enum": ["\/contexts\/Organisation"],
            "required":true
        },
        "@id": {
            "type": "string",
            "pattern": "^\\/organisations\\/[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$",
            "required":true
        },
        "@type": {
            "enum": ["Organisation"],
            "required":true
        },
        "title": {
            "enum": ["Acme"],
            "required":true
        },
        "street": {
            "enum": [null],
            "required":true
        }
    },
    "additionalProperties": false
}
"""

which you could also write like this:

And the JSON should be valid according to this schema:
"""
{
  "$schema": "http://json-schema.org/draft-03/schema",
  "required": true,
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "@context": {"required":true, "enum":["\/contexts\/Organisation"]},
    "@id": {"required":true, "type":"string", "pattern":"^\\/organisations\\/[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$"},
    "@type": {"required":true, "enum":["Organisation"]},
    "title": {"required":true, "enum":["Acme"]},
    "street": {"required":true, "enum":[null]}
  }
}
"""

which is even closer to your "equal to considering regular expressions" (but still more verbose, yes). There's a difference though: the schema doesn't enforce the order of properties, while equal does.

[Toflar]

Yeah, my !regex: solution has turned out to work perfectly fine for mulitiple projects. It's very simple and powerful.

[guilliamxavier]

Do you mean that you eventually implemented it? (I couldn't find any PR from you)

[Toflar]

Yeah I've added a custom step in my own context to do this and it works but it's actually a chunk of ugly code that would require polishing, testing etc. I don't currently have the time to work on this but I'll just leave the code here. Maybe someone likes my idea, needs it to for their own projects and wants to bring it to behatch in a nicer, cleaner way 😄

There you go:

    /**
     * @Then the JSON should be equal to considering regular expressions:
     */
    public function theJsonShouldBeEqualToConsideringRegularExpressions(PyStringNode $content)
    {
        $expected = json_decode((string) $content, true);
        $result = json_decode((string) $this->httpCallResultPool->getResult()->getValue(), true);

        if (!$this->checkIfTwoArraysHaveSameStructure($result, $expected)) {
            throw new \Exception(
                sprintf('Cannot compare regular expressions as the JSON does not even have the same structure (must already fail!). Result was "%s", expected was "%s".',
                    json_encode($result),
                    json_encode($expected)
                ));
        }

        $expected = $this->recursiveHandleRegexMatching($expected, $result);

        $this->assert($result === $expected, sprintf(
            'The JSON result "%s" does not match the expected "%s".',
            json_encode($result),
            json_encode($expected)
        ));
    }

    private function checkIfTwoArraysHaveSameStructure(array $array1, array $array2): bool
    {
        if (0 !== \count(array_diff_key($array1, $array2)) || 0 !== \count(array_diff_key($array2, $array1))) {
            return false;
        }

        foreach ($array1 as $k => $v) {
            if (\is_array($v)) {
                if (!\is_array($array2[$k])) {
                    return false;
                }

                if (!$this->checkIfTwoArraysHaveSameStructure($v, $array2[$k])) {
                    return false;
                }
            }
        }

        return true;
    }


    /**
     * @throws Exception
     */
    private function recursiveHandleRegexMatching(array $expected, array $result): array
    {
        if (0 === \count($expected)) {
            return $expected;
        }

        foreach ($expected as $k => $v) {
            if (\is_string($k) && false !== strpos($k, '!regex:')) {
                throw new Exception('Cannot assert regex in JSON keys.');
            }

            if (\is_array($v)) {
                $expected[$k] = $this->recursiveHandleRegexMatching($v, $result[$k]);
                continue;
            }

            if (null === $v || !\is_string($v)) {
                continue;
            }

            preg_match_all('/!regex:((?:[^"\\\\]|\\\\.)*)/', $v, $matches, PREG_OFFSET_CAPTURE);

            if (0 === \count($matches[0])) {
                continue;
            }

            $rgxp = $matches[1][0][0];
            $offset = $matches[0][0][1];

            // Assert the match fulfills the regex
            if (1 !== preg_match($rgxp, $result[$k])) {
                throw new \Exception(sprintf(
                    'The regex "%s" does not match value "%s".',
                    $rgxp,
                    $result[$k]
                ));
            }

            // Replace inner match with original content to then globally compare
            // if the rest of the content matches
            $start = substr($v, 0, $offset);
            $end = substr($v, $offset + \strlen($rgxp) + 7);  // 7 = !regex:
            $v = $start.$result[$k].$end;

            $expected[$k] = $v;
        }

        return $expected;
    }

    /**
     * @throws Exception
     */
    private function assert(bool $test, string $message)
    {
        if (false === $test) {
            throw new \Exception($message);
        }
    }

[guilliamxavier]

Thanks for sharing anyway 🙂