Spec Conformance Tool - Service Validator

[leecalcote]

Reliable specifications (protocols) require validation and testing to guarantee interoperability, particularly as a specification evolves (versions over time). We're in need of a tool (maybe two?) to validate event formats of operational services to verify their conformance to the normative statements in the CloudEvents specification. To facilitate validation, a checklist of assertions (as a document separate from the CloudEvents spec) may be needed.

Both the conformance tool and list of assertions (requisite tests) should be versioned alongside the CloudEvents specification.

This "CloudEvents Service Validator" would ideally be an open source framework for checking conformance of any generic service implementing the CloudEvents format as defined in the CloudEvents specification. The tool is designed to be vendor/project agnostic and driven purely based on the CloudEvents specification.

Integrity of validation results need to be guaranteed. Therefore, hosting of such a small service validator may be required.

[bjartek]

Looking at https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.2/tck/README.md could be a good idea here.

[duglin]

There's this: https://github.com/btbd/CEVerify in our list of open source projects. @leecalcote what are you thoughts on where to take this next?

[jroper]

I agree that a TCK would be great - a TCK isn't just useful for validating implementations, it would be useful for validating the spec itself. I think there are a number of TCK tests we could write at this point that would prove the spec had bugs in it, such as the JSON encoding bug that would see Any typed attributes with a Binary value coming out at the other end as a Base64ed String value, not the original Binary value.

CloudEvents of course is an abstract concept, meaning that you can't write a TCK like the Reactive Streams one mentioned above - there's no application API that can be verified. But, the transport bindings and formats are concrete, and can be verified.

So, I would suggest two types of tests in the TCK. The first is round tripping a single format/transport. So, for example given a JSON encoded cloud event, an implementation should parse it, and then output the same CloudEvent encoded as JSON. The TCK should then check that the input and output JSON are semantically identical. If it's not, the test fails.

The second type of test is conversion. So, for example, the TCK may pass a Cloud Event using the HTTP binary transport mode to the tested implementation. That implementation must then output the CloudEvent as JSON (not necessarily in a response, but the TCK binding for the implementation will retrieve the CloudEvent from wherever the implementation put it, and pass it back to the TCK for verification. The TCK will then check that the returned JSON encoded CloudEvent is semantically identical to the HTTP encoded CloudEvent that it submitted. Multiple permutations of this could be defined, eg HTTP to Kafka, Kafka to AMQP, JSON to Protobuf, etc. Implementations won't implement all of these, but can run whatever TCK tests for the permutations that they do support.

We would need to decide on a programming language to use to implement the TCK.

[duglin]

FYI http://soaphub.org/ceverify/ has been updated to support/test/verify the latest CE spec

[duglin]

I"m going to suggest that we close this - does anyone have any concern about this?

[leecalcote]

Thumbs up here.

On a related note, the Meshery project (https://meshery.io) has begun using CloudEvent compatible messages between some of its components. As fate has it, Meshery also provides conformance validation for the SMI specs: https://meshery.io/smi.

Random trivia.

[jskeet]

Not convinced on closing this one - I think some sort of conformance tool would be useful.

For example, I have a bunch of tests around HTTP header value encoding and decoding from back when I revisited that part of the spec. If we had those in a platform-neutral format so that all SDKs could use them, that would be good.

I acknowledge that scoping and designing the TCK is tricky here - I tend to think of it as a TCK for SDKs, but that's because I'm an SDK maintainer. Maybe one TCK for SDKs, another for Discovery etc is needed. Designing the TCK for SDKs is likely to be tricky due to the significant variation in how things are handled - but I'd quite like to keep this issue open to acknowledge the value in such a TCK existing if anyone ever has enough spare time to create one :)

[pcgeek86]

I agree that a validation tool of sorts would be useful, to ensure interoperability.