Multi tenant emulator

[bluesign]

Closes #207

Description

This is a small PR allowing using HTTP header Emulator to quickly start a new blockchain instance and switch between them with Emulator: [someIdentifier]

This is also on the idea phase, and we can discuss over it, but I think it can be very useful, as flow-js-testing etc, always launching new emulator instances, starting and stopping emulators.

This can also let few users to use a single emulator server concurrently. (no port clashes etc)

---

For contributor use:

• [X] Targeted PR against master branch
• [X] Linked to GitHub issue with discussion and accepted design OR link to spec that describes this work
• [ ] Code follows the standards mentioned here
• [ ] Updated relevant documentation
• [X] Re-reviewed Files changed in the GitHub PR explorer
• [ ] Added appropriate labels

[turbolent]

What are the advantages of this built-in support for multiple emulators compared to starting multiple emulators on different ports?

[bluesign]

What are the advantages of this built-in support for multiple emulators compared to starting multiple emulators on different ports?

@turbolent

2 big advantages:

• It is cheaper than launching emulator; especially when resource-constrained conditions ( like CI running tests or wasm emulator on the web )

• No need for heavy emulator management. For example, in flow-js-testing: There is a need for big management code they have to write: They need to find empty ports, launch the emulator with them and switch the FCL config to that port ( but as the config is singleton there, they need to run tests inband, not parallel, etc.)

Actually, this idea came from trying to speed up flow-js-testing. ( and run tests in parallel easily )

[janezpodhostnik]

I see the utility of this when running in a test setting.

Not sure if using headers to switch versions would be my first choice. Maybe it would be easier to just bind multiple ports each to their own emulator (own blockchain, not own emulator process).

With the current code I'm also concerned about concurrent access. Can we add some tests?

[bluesign]

@janezpodhostnik thank you very much for the comments, this is to kick off the discussion PR (issue didn't get much attention I wanted to show what I mean more clearly ), I was gonna make this draft PR, but somehow I am GH cli noob. I will add tests and concurrency improvements for sure.

For the headers part: This is the standard API key model, but I just recognized that I didn't give enough context within the issue and PR about the direction I am targeting. So let me briefly explain what I have in my mind.

The emulator is the core component when doing the development. There are a few pain points I recognized where users always struggle when developing and testing dapps, such as:

• slow testing ( especially with js-testing )
• state management ( we have some state management, but I am not proud of it )
• different cadence versions ( especially testing with upcoming cadence versions on breaking changes )

For the first target ( slow testing ), multiple ports are a solution. Eventually, it is a bit tricky and not sustainable in scale ( think of the playground using a backend for many users/projects). In that direction, with new storage backend improvements https://github.com/onflow/flow-emulator/pull/221 I am trying to find a suitable backend that can scale and with https://github.com/onflow/flow-emulator/issues/219 trying to make it cost-viable. ( I have a few playground alike projects like jupyter notebook etc.)

For the state management, this provides at least context switching for free, forking part has to be worked on for sure.

For the different cadence versions, it is a bit of another direction ( or a long-term goal ); with these changes, it leads to the possibility of running an emulator as a SaaS ( think of it as a disposable emulator server where you can connect to an instance of the emulator by pointing out your client with an API key + project identifier ) and run your tests. So basically, just changing the endpoint, you can run it all on a different instance of the emulator running with the upcoming version of cadence.

[devbugging]

@bluesign it feels to me like this could/should be handled on another layer, not by the emulator itself. I'm afraid it adds complexity in the codebase that is very specific to running tests maybe. I understand the pain point is bootstrapping the emulator so wouldn't a solution potentially be having a pool of bootstrapped emulator instances and then doing some kind of load balancing between them where there can be a policy on the load balance to know which instance it forwards to. I'm a bit on the fence with this one tbh. So can we try and look into any solutions on another layer if we can avoid adding the complexity of this here, especially since I'm not too convinced this is a common problem yet.

[bluesign]

I understand the concerns, and to be honest, I don't have too hard an opinion on this feature too. We can maybe drop this PR. My only concern was every consumer ( js-testing, overflow, playground ( my version ) ) has to manage same thing themselves.

[devbugging]

I understand the concerns, and to be honest, I don't have too hard an opinion on this feature too. We can maybe drop this PR. My only concern was every consumer ( js-testing, overflow, playground ( my version ) ) has to manage same thing themselves.

But is that the current issue with overflow and playground? For playground I can say that we use emulator as a module and have instances bootstrapped as part of the instance pool so we can always have them ready: https://github.com/onflow/flow-playground-api/blob/master/blockchain/pool.go#L28 I will move this to draft for now ok? maybe we can revisit if more usage will require this but for now I'm still not convinced it justifies the added complexity and possible issues with maybe some concurrency.