Support for realms/contexts in `deno_core` (in preparation for `ShadowRealm`)

[andreubotella]

In JS, code execution is always tied to some realm (or "context" in V8 terminology), which provides a global object and a set of built-ins (Object, Array, Error, etc.) which is different from those of other realms. Multiple realms can share an event loop (= a thread), and those that do share one may be able to access objects from a different realm. In the web, a same-origin <iframe> is a different realm from the top-level page (iframe.contentWindow.Array !== window.Array). Currently there is no way to create realms or run code in them purely with JS built-ins, that's something that host environments (the web / Node.js / Deno) would have to provide, so it's fine for Deno to not support them.

But there is currently a stage-3 TC39 proposal called ShadowRealm that would allow creating realms, as a way of sandboxing untrusted code. This proposal is not close to shipping anywhere, and it's only now starting to get implemented in V8, but currently Deno assumes that there's a single V8 context available at all times, and it's better to start refactoring things to support realms with plenty of time to spare.

Since ShadowRealms are meant to be a sandbox primitive, there would be no way to make objects from the parent realm available inside the ShadowRealm and vice versa. This simplifies to some extent the requirements needed, because that way there is no need to comb through the JS code looking for wrong uses of instanceof. That said, these are the requirements that seem to be needed for ShadowRealm (and I'm sure I'm forgetting some):

• [x] Add a Rust API to create V8 contexts associated with a JsRuntime and run scripts in them (similar to JsRuntime::execute_script), to simplify running tests. Modules can be left for later. #14019
• [x] Make sure bindings and extension scripts are initialized correctly in the context, both when using snapshots and when not. This includes making sure the ops cache in different realms can't go out of sync. #14019
• [ ] Make sure any object return values of bindings and ops, as well as any exceptions they throw, are objects in the current realm (i.e. their prototype is not, say, a different realm's Object.prototype).
• [ ] Make sure async ops don't resolve a promise in the wrong realm. #14734
• [ ] js_promise_reject_cb, js_uncaught_exception_cb and js_wasm_streaming_cb should probably be realm → callback (weak)maps.
• [ ] Modules. Per the HTML spec, every realm has its own module map, although module fetches can be cached across realms.
  • [ ] Add Rust APIs to load modules in a context (similar to JsRuntime::load_main_module and JsRuntime::load_side_module).
  • [ ] There should be a module map per realm, rather than one for the entire JsRuntime. (Should there be a single module loader, though? Blobs should probably not be cached across realms, but non-blob fetches probably should, so it's not clear what to do about #12458.)
  • [ ] Make sure module evaluation in realms integrates with the event loop.
  • [ ] Dynamic imports and import.meta.
• [ ] Review every use of serde_v8::Value to make sure objects from a realm can't leak into another.
• [ ] Fix the inspector to properly support multiple realms. #14046

[andreubotella]

For the general case with arbitrary realms, not just those with the security properties of ShadowRealm, there are more requirements and questions:

• Comb through the JS runtime code to make sure instanceof checks are sound in the presence of objects from a different realm.
• HTML and WebIDL have convoluted requirements regarding in which realm JS code gets executed, and in which realm the return values of web APIs are constructed. (I confess I don't fully understand those myself.)
• serde_v8 and the bindings must support inputs from a different realm.
• Creating contexts with an empty global, even though the isolate was created from a non-empty snapshot. Also, creating contexts from arbitrary existing objects. (Both needed to implement Node.js's vm module.)

[bartlomieju]

Inspector support for multiple contexts should also be considered when working on this feature. What you already proposed is gonna be rather large undertaking, I expect even more non-trivial engineering effort for inspector integration.

[andreubotella]

Recently denoland/deno#13861 has radically changed how ops work internally, such that op synchronization is no longer needed. Also, denoland/deno#13993, although still experimental, should guarantee that async ops resolve the right promise on the right realm. Both of this PRs simplify the work needed for refactoring deno_core to support realms, and as a side effect, reduce the number of JS callbacks that need to be kept in a per-context weakmap.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[andreubotella]

Not stale. This is being worked on, slowly.

[andreubotella]

Status update: in October, many of the PRs that I landed related to realms were reverted in denoland/deno#16366 because they were blocking some planned optimizations in deno_core. The plan was for those optimizations to take about two weeks to be implemented, and to start working on relanding those patches afterwards. Those optimizations took way longer than that, but I am now starting to work on relanding those PRs.

I have updated the status of the bullet points in the OP to mark whether they are currently reverted or relanded.

[bartlomieju]

@andreubotella can we close this issue now?

[andreubotella]

I intended this to be a meta-issue covering everything up to shipping ShadowRealm, and although realms now work fine for deno_core embedders, there's still some work needed before the ShadowRealm constructor can be implemented (see denoland/deno_core#101). And after that there's still the work of properly boostrapping the realm in deno_runtime and integration testing.

[bartlomieju]

Sounds good, let's keep it open then.