defradb icon indicating copy to clipboard operation
defradb copied to clipboard

A bug in testing framework if error is raised and assertion of the error passes, it omits other assertions

Open shahzadlone opened this issue 1 year ago • 7 comments

Describe the problem Our testing framework seems to be not asserting the other result values once an error is encountered and assertion for that ExpectedError passes.

To Reproduce Both the below tests pass (Results field in each is different):

func TestStopsAssertionOnceRaisedErrorIsAsserted1(t *testing.T) {
	test := testUtils.RequestTestCase{
		Description:   "Delete multiple documents that exist sub selection, should give error.",
		Request:       `mutation { delete_user(ids: ["bae-xyz"]) }`,
		Results:       []map[string]any{{"_key": "Doesn't assert the result if raised expected error."}},
		ExpectedError: "Field \"delete_user\" of type \"[user]\" must have a sub selection.",
	}
	executeTestCase(t, test)
}

func TestStopsAssertionOnceRaisedErrorIsAsserted1(t *testing.T) {
	test := testUtils.RequestTestCase{
		Description:   "Delete multiple documents that exist sub selection, should give error.",
		Request:       `mutation { delete_user(ids: ["bae-xyz"]) }`,
		Results:       []map[string]any{},
		ExpectedError: "Field \"delete_user\" of type \"[user]\" must have a sub selection.",
	}
	executeTestCase(t, test)
}

Expected behavior The test should assert other fields (like Results in this case) in this case, and ensure tests only pass if all testing args are what we expect them to be, which doesn't make sense so preferably not allow this at all in the testing framework.

shahzadlone avatar Apr 16 '23 12:04 shahzadlone