Guidance/Tooling for Testing?

[michaelpnash]

It would great to have a specific set of recommendations to Cloudstate users for how to test their functions, especially when it comes to testing with the Proxy itself.

I assume unit testing would follow whatever is normal for the development language, but what would we need (and would we need specialized tooling to help) as far as integration testing with Cloudstate?>

[sleipnir]

Hello @michaelpnash I think the first step would be to provide a way to run the proxy locally and with a transparent network between it and the user's function. So I added this feature to the CLoudstate Community CLI and in the Cloudstate previous issues the disussion can be found here. After that we would have to create a way to automate requests from the user's entity classes. We could find out via proto definition file which entity service operations can be called and generate the necessary stubs (akka-grpc-plugin should be useful here), packaging all this in a 'Cloudstate' plugin maven and gradle may be useful.

This is basically the same strategy that I use to test the shoppingcart from the Kotlin repository https://github.com/cloudstateio/kotlin-support/blob/master/cloudstate-kotlin-support/src/test/kotlin/io/cloudstate/kotlinsupport/tests/IntegrationTest.kt

[michaelpnash]

@sleipnir Yes, agreed, a way to run the proxy and test against it is likely a large part of this. @jroper I believe you had a design in mind here?

[jedahu]

Created a new issue #318 to track implementing the above (run the proxy and test against it).

[jedahu]

Initial thoughts:

Create a language independent way of describing requests and expected responses. This will likely take the form of a schema for a well known data format like JSON. Ideally implemented such that real-world payloads can be copied into this description with minimal ceremony, to facilitate creating test cases.

Create a way to "run" the request+response list against a user function. Something like the TCK but for users rather than language front-end implementers.

Equality could be handled by the test runner by reflecting on the rpc types. Perhaps the request+response description could provide a way to override this?

Also need to consider how to parameterize the tests or if this is even necessary.

[jroper]

Integration tests I think would also be language dependent - otherwise if we create some DSL for writing integration tests, we're effectively creating a new language that users have to learn and use for writing their integration tests, that's not going to stick, if I'm a Java developer, I want to write my integration tests in Java using JUnit (or TestNG or whatever), while if I'm a JavaScript developer, ~~I don't know what tests are so this issue is irrelevant~~ I want to write my integration tests in JavaScript using mocha or whatever this weeks JS test framework is.

I think this task is almost all about how to run the proxy (or a mocked version of it) locally. It may involve writing some tooling, such as build plugins, but that's going to be specific to a language. It may involve writing some test harnesses to be able to interrogate what data was stored, or to be able to run assertions on events that were emitted, etc, some of this might live in the mock proxy, while perhaps language specific clients to consuming that data might be provided as well.

[marcellanz]

@jedahu I tried a bit going in this direction using https://cuelang.org hat can import protobuf files to cue schemas, also import JSON files and validate "structures" by a schema defined in CUE. I also have used the proxy from grpc-tools to capture real TCK traffic and then imported the dumped JSON to create a simple load test as an example.

CUE recently added property attributes which may help to have an "independent way of describing requests and expected response" where a complementary tool would interpret instructions like sending a message or validating responses.

To be clear, I use this for traffic between the proxy and the user function and I don't know yet how complete a schema in CUE can be to describe the Cloudstate protocol and validate user languages against, my main motivation doing this. While cue knows how to import proto files and JSON files and validating them, it would need for sure a complementary runner to load cue definitions and interpret a request-response DSL.

Using cue for this might be too esoteric or the wrong tool to do it. I found to have a protobuf file be imported to cue, safe validation, an API and also exports to JSON, YAML or even OpenAPI interesting.

---

This is a way to describe a shopping cart get init-ed and then send a few remove_add sequences. The cue export output of this can be feeded to grpc-replay and a ShoppingCart TCK would respond.

remove_add :: RemoveLineItem + AddLineItem
sequence : msg : EventSourced_RPC & {
	messages: EventSourced_init_and_add.messages + (100_000 * remove_add)
}

RemoveLineItem: [{
	message_origin: "client"
	raw_message:    "GnYKCnRlc3R1c2VyOjIQBxoKUmVtb3ZlSXRlbSJaCjt0eXBlLmdvb2dsZWFwaXMuY29tL2NvbS5leGFtcGxlLnNob3BwaW5nY2FydC5SZW1vdmVMaW5lSXRlbRIbCgp0ZXN0dXNlcjoyEg10ZXN0cHJvZHVjdDox"
	message: command: {
		entityId: "testuser:2"
		id:       "7"
		name:     "RemoveItem"
		payload: {
			"@type":   "type.googleapis.com/com.example.shoppingcart.RemoveLineItem"
			productId: "testproduct:1"
			userId:    "testuser:2"
		}
	}
	timestamp: "2020-05-01T15:44:10.129917+02:00"
}

[michaelpnash]

@marcellanz If we put together what you've described here with James's thought of a "Mock Proxy", I could image that mock taking a sequence, and interacting with the user service, running through that sequence in a "send-expect" fashion. If the mock proxy (which we would have to call "moxy") was in a docker container, it might be more language-independent (that is, if you are a JS/TS developer and don't want to have to have a JVM on hand, you just run the docker image).

Then "success" of the test is the sequence succeeding (got all the events I expected to the events I sent)...

This sounds like something I'd be very happy using to test my functions and their interactions with the proxy, short of actually deploying it.

[marcellanz]

@michaelpnash Thinking about this made me curious what a good UX would be for a developer.

If we say that the user can focus on its business code, learning how to define gRPC messages in a text file is perhaps a lot to be expected? Using grpcurl could even be less intimidating than a CUE definition file. What I wrote above sounds practical at first, but thinking more about it, I'm not sure anymore. At least for the user facing developer.

If I imagine to be a developer living in my Language-X world, a send-expect test written in this language and watching, not writing, JSON representations of my calls is perhaps the more comfortable way? For the developer the gRPC service he defined has to be implemented in a special way, the way we define for language supports for Cloudstate and that one is different what he might know if he knew how to implement normal gRPC services before.

When it comes to debugging or defining special sequences perhaps between the proxy and the user function or defining (by spec) the Cloudstate Protocol, the low-level approach above might be more helpful I think.

Regarding mocking the moxy-proxy, If I could choose, I'd like to run a local "Cloudstate Engine" and have a real proxy running locally. wdyt?

[michaelpnash]

@marcellanz Indeed, it is important to have a test process that is as familiar and comfortable to the developer in their own language as possible. We've found grpcurl gets used quite a bit when experimenting, but send-expect in the developers lang of choice sounds even better.

If we had a proxy that is, in virtually all respects, the same as a "real" proxy, but packaged into a Docker container (so I don't need to build it or even have a JVM on my machine), then perhaps we could "drive" that proxy to go through specific sequences, e.g. "proxy, please send this to my service, and expect this in return", where that is expressed in the language of choice. This would require some manner of polyglot adapter, though, to turn that into gRPC requests/responses to the "special" proxy. It might also be helpful to be able to validate against the journal, e.g. "send these requests, expect these response, then there should be entries in the journal that look like this", essentially.

Even as I describe it, this is perhaps still too complex :-)

Running a "real" proxy locally feels a bit heavyweight, though, especially if that real proxy needs a real data store. At some point the test process should encourage the developer to simply deploy to an appropriate cluster, and test there, I suspect, as a "full" integration test - it's the steps before that where I believe we could help more.

[sleipnir]

@marcellanz @michaelpnash I may be mistaken, but I believe that we already have the "real" proxy running via docker, perhaps what we need is an "interceptor app" in the middle of the path that is able to validate the inputs and outputs of the existing proxy.

[marcellanz]

@sleipnir yeah, I use the dev-mode proxy a lot, it starts in 2-3 seconds and does not join any other proxy by its dev-mode configuration.

[marcellanz]

@michaelpnash wrote:

Even as I describe it, this is perhaps still too complex :-)

@michaelpnash @sleipnir so perhaps we can describe that story!?

For discussion and if it is given that the developer has

1. a gRPC service definition the User Function A => a_function.proto
2. Entities E1…En for A implemented in Language X => E1_entity.x, …, En_entity.x
3. a Client that connects to the Proxy and interfaces the User Function.

that connects like

[ Client ] <--(a)--> [ Proxy ] <--(b)--> [ UserFunction ]

with:
(a) => gRPC defined by 1.
(b) => Cloudstate Protocol

• The Proxy discovers any services from the UserFunction and exposes them to a Client.
• The UserFunction chooses a state model for an entity and registers that to its Cloudstate user support library.
• The UserFunction implements the entity with command handlers for the state model choosen.

to have a baseline, from here on, what would function testing tooling do for the developer?

What does it mean when the developer "instructs the proxy" to send message-1 (request) and then expects message-2 (reply)? What would that sequence be different by having a gRPC client stub of the defined UserFunction in 1. and use the stub to send these messages? Why is the developer interested in any messages on (b) between the Proxy and the UserFunction?

For (b) the UserFunction receives two kind of messages j) any message defined for the UserFunction in 1. through Cloudstate Protocol Command messages. k) messages from the proxy sent to support the the state model defined by the Cloudstate Protocol.

For k) depending on the state model we would see:

• side effects
• init messages (EventSourcedInit)
• event messages (EventSourcedEvent)
• commands
• snapshots
• client actions (evensourcing: reply, forwards, failures)
• and also sequence numbers for these messages
• ...

We could define a simple well known usecase and then derive what is needed from Tooling to support testing.

A few loosely thoughts/ideas:

• message templates for the messages in k) that then get completed by concrete values?
• an API/script/definition to instruct the proxy? like 1 proxy.send( Command("add", "entity-1", "apples") ) 2 proxy.send( Command("add", "entity-1", "oranges") ) 3 proxy.crash( "entity-1" ) 4 proxy.expect( InitMessage ("entity-1", ["apples", "oranges"]) ) 5 proxy.send( Command("add", "entity-1", "bananas") ) 6 proxy.send( Command("get", "entity-1")) 7 proxy.expect( "entity-1", ["apples", "oranges", "bananas"] )
  • 4 is a k-message, one that a user never sees normally, but has to implement for snapshots or crdt init-messages.
  • Functionally, if a developer can verify 7) he does not need to know that snapshot messages where sent to restore state on 4), or should he?

With that above, perhaps we can get a story to capture requirements and then decide how to implement these requirements by the needs expressed.

wdyt?

[sleipnir]

@marcellanz I thought the suggestion was very good, I just don't know if I understood correctly what will be exposed to the end user

[jroper]

On the topic of whether it's a real proxy or mock proxy, it's kind of both. So, it will be a build of the real proxy code, but not the same build that is used in production, here are the differences:

• The real proxy, when it starts up, attempts to contact Kubernetes to perform cluster bootstrap - obviously, a proxy that a user runs on their machine is not going to do that, instead it will bootstrap a cluster of one node with itself. This is what the devmode build of the proxy does.
• The mock proxy may be configured to use an in memory database, or perhaps an hsql or leveldb database that writes its store to disk, which could be a volume mounted from the developers host machine, so that data survives restarts of the proxy.
• We may decide to handle eventing differently - since locally, a user might not be running a message broker. Instead, the eventing in the mock proxy may expose an additional interface for sending events, and it may buffer the events it emits in memory, allowing them to be received through that interface.
• We may decide that it's useful, at this level of testing, to still be able to interrogate the datastore directly - so the proxy may offer an interface for asking for the raw events and raw data stored.
• A number of production features would be disabled - this might include metrics gathering, things related to autoscaling, etc.
• Some future features may need to be done differently, eg, one possibility in future is that we may have some support for authentication and authorization built into the proxy, this would likely require a different implementation and perhaps some level of direct external control for a locally run proxy.

As far as what the tests actually look like, I would expect that users would simply use their own generated gRPC client, or their own REST client, to make calls on the proxy. We wouldn't provide any tooling for that, the point of tests at this level is that they use the public interface exposed by the proxy, otherwise why have the proxy at all? Indeed, users might be writing integration tests where their test code doesn't talk to the proxy, but rather another service that they're also testing is calling the proxy. We would only provide the necessary tooling for starting up and configuring and managing the lifecycle of the proxy, along with APIs for eventing and interrogating the datastore as described above. But for the most part, once set up, the tests will just be black box tests that use the public interface exposed by the proxy.