Browser-Only: Run Theia without a backend

[sdirix]

Current state

• The foundation for this feature was merged with #12853
• We are still missing a lot of features, contributions are welcome!

File management

• [ ] Finish the BrowserFS filesystem provider
  • [ ] Implement file watching
  • [ ] Implement directory deletion
  • [ ] Implement file renaming
• [ ] Missing browser-only feature: Import Files into workspace (context menu > Upload files…)
• [ ] Missing browser-only feature: Download files from workspace (context menu > Download)
• [ ] Missing browser-only feature: Search in files

Features

• [ ] Support Theia plugins and VS Code web extensions
• [ ] Support user preferences
  • Workspace preferences are already supported via BrowserFS filesystem
• [ ] Adapt remaining pieces of preloader
• [ ] Support terminal / console
• [ ] Support localization
• [ ] Support source control
• [ ] Support debugging
• [ ] Support OpenVSX registry
• [ ] Support tasks
  • Or remove feature if not applicable
• [ ] Show list of extensions in about dialog
• [ ] Support log levels
• [ ] Support theia test for browser-only targets

Testing

• [ ] Implement UI smoke tests which run regularly

Documentation

• [ ] Document on Theia website

tl;dr;

We would like to enable a new use case for Theia applications: Support for "browser-only" Theia applications which can be executed without a backend.

Our vision is to support Theia "browser-only" applications as a first class citizen, next to the "browser" and "electron" targets. For this we adapted the Theia CLI tooling to support the new browser-only target and frontendOnly Theia extensions. Using this tooling the feature can be supported in a minimally invasive way, requiring none (or little to none) changes to existing Theia based applications.

Check it out yourself

Because it's so easy to host static sites, we deployed three example applications for you to check out:

• Minimal browser-only application
• Including all Theia packages
• POC for VS Code web extension support
  • You can execute a hello world command implemented by a Theia plugin
  • By opening a Typescript file you can see the Typescript VS Code web extension at work
  • You can clone Github repositories via the File > Import repository action. Note that this uses the open proxy cors.isomorphic-git.org so it's pretty slow. This proxy is freely available so please be considerate with its usage.
  • Note that this POC is not optimized, so it takes a while to start.

Details

Click here for the original issue text including high level overview of the feature

Motivation

Support of browser-only use cases is a value adding feature for the whole Theia ecosystem. While the existing Theia framework offers large flexibility and therefore supports a wide array of use cases, it requires the deployment and execution of a Node based backend. This often comes with a lot of operational effort and additional costs.

There are many use cases in which not the full feature set of Theia based applications are required. For example, demo and quick start applications often come with hard coded examples anyway, do not require persistent storage and can work without terminal support.

This feature is requested from time to time in the community too:

• https://github.com/eclipse-theia/theia/issues/8368
• https://github.com/eclipse-theia/theia/issues/6732
• https://github.com/eclipse-theia/theia/issues/6560
• https://community.theia-ide.org/t/does-theia-work-without-any-backend/46
• https://community.theia-ide.org/t/would-it-be-possible-to-create-a-backend-less-theia/1545
• https://community.theia-ide.org/t/can-i-build-theia-as-a-browser-front-end-widget-only/646
• https://community.theia-ide.org/t/client-only-ide-for-typescript/1622

Implementation

Theia's tooling already generates the frontend separately from the backend in the lib/frontend directory. So technically the frontend can already be served with any web server. However, the frontend will try to establish a web socket connection to its backend and many services will not work when their proxy counterparts are not available.

Tooling changes

We added a new target for Theia applications: browser-only, e.g

  "theia": {
    "target": "browser-only"
  }

We added a new entry for Theia extension modules: frontendOnly, e.g.

  "theiaExtensions": [
    {
      "frontend": "lib/browser/i18n/i18n-frontend-module",
      "frontendOnly": "lib/browser-only/i18n/i18n-frontend-only-module",
      "backend": "lib/node/i18n/i18n-backend-module"
    },
  ]

For this target, the Theia CLI (theia build) will:

• skip generating the backend
• for src-gen the frontend/frontendOnly pairs are handled in the same way as backend/backendElectron are, i.e. if both are present, frontend will be skipped.

Using these new tools we were able to replace, mock and/or remove all services which need to connect to the Theia backend.

theia start is supported for browser-only targets too: It will launch the frontend with the webpack-dev-server.

Implementation changes

Using the tooling described above we were able to support the new browser-only target almost without any changes to the existing code base. The current PR only contains two breaking changes: A rename of a class, and the move of some classes from the node directory to the common directory. All other changes are only contained within frontendOnly modules.

The following subsections describe the most important features:

Filesystem

Since there is no native file system available in the browser, we settled on using BrowserFS, as it offers a Node compatible API, is feature rich and is well supported.

We foresee that every browser-only Theia application might want to handle the filesystem in a different way, for example to exclusively bundle some hard coded files, handling persistence via webworker cache, local storage, IndexedDB or only in memory. Therefore, we added a new service, the BrowserFSInitialization, that can be customized to support any behavior.

In the PR, we used this service in the new "browser-only" example application to initialize the file system with some basic example JSON files.

WebSocketConnectionProvider

The WebSocketConnectionProvider of Theia is used to create proxy services for Theia's RPC mechanism.

Via our proposed PR, we already support a minimal Theia without using any proxy services consisting of @theia/core, @theia/filesystem, @theia/editor, @theia/getting-started, @theia/keymaps, @theia/markers, @theia/monaco, @theia/navigator, @theia/outline-view, @theia/preferences, @theia/property-view, @theia/userstorage, @theia/variable-resolver and @theia/workspace.

To enable using all Theia packages, even if they still depend on WebSocketConnectionProvider, we rebind WebSocketConnectionProvider to a mock implementation whose connections will always timeout. This allows using all Theia packages without further changes, even though the functionality will be broken wherever it relies on the backend, e.g. the file search will never complete.

Eventually we would like to see all Theia packages supporting their full feature set (as far as possible) for the browser-only case as well. Once that is the case, we technically no longer need a mocked WebSocketConnectionProvider in the browser-only frontend, however it might make sense to still keep it for 3rd-party dependencies which are not adapted yet.

Outlook

For performance reasons it could make sense to move some services to web workers to lessen the load on the main Javascript thread of the frontend. For this use case an alternative ConnectionProvider could be implemented which bridges the gap to the web worker instead of to the backend.

Open Tasks

We think it already makes sense to merge this feature into the Theia code base via this PR. It's already possible to create specific feature-complete browser-only applications for certain use cases. The situation will only become better with more features being iteratively enabled over time. In communication with the community it must be made clear that a "browser-only" Theia can't support all use cases of a regular "browser" or "electron" Theia and that for now this feature shall be considered experimental.

We collected a non-exhaustive list of tasks which can be tackled as a follow up of the current PR.

File management

• Finish the BrowserFS filesystem provider
  • Implement file watching
  • Implement directory deletion
  • Implement file renaming
• Missing browser-only feature: Import Files into workspace (context menu > Upload files…)
• Missing browser-only feature: Download files from workspace (context menu > Download)
• Missing browser-only feature: Search in files

Features

• Support Theia plugins and VS Code web extensions
  • for more information see the following section
• Support user preferences
  • Workspace preferences are already supported via BrowserFS filesystem
• Adapt remaining pieces of preloader
• Support terminal / console
• Support localization
• Support source control
• Support debugging
• Support OpenVSX registry
• Support tasks
  • Or remove feature if not applicable
• Show list of extensions in about dialog
• Support log levels
• Support theia test for browser-only targets

Documentation

• Add documentation for browser/browser-only modules in https://theia-ide.org/docs/architecture#separation-by-platform

Theia plugins and VS Code web extensions support

The largest missing piece is Theia plugins and VS Code web extensions support.

The good news is that we already have a POC showcasing that the support of Theia plugins and VS Code web extensions is conceptually feasible:

• You can find the hosted POC here
  • You can execute a hello world command implemented by a Theia plugin
  • By opening a Typescript file you can see the Typescript VS Code web extension at work
  • You can clone Github repositories via the File > Import repository action. Note that this uses the open proxy cors.isomorphic-git.org so it's pretty slow. This proxy is freely available so please be considerate with its usage.
  • Note that this POC is not optimized, so it takes a while to start.
• You can find the code on a branch here

Conceptually, Theia is well prepared to support Theia plugins and VS Code web extensions in the browser-only case, as these are already executed in the browser in a webworker.

However, it will require some substantial effort until plugin support will be ready for production in the browser-only case:

• All plugins are organizationally exclusively handled on the backend side, especially metadata collecting and deploying
• Any call to the Theia API / VS Code API which is currently routed through the backend will not work in the browser-only case

Even though there is no obstacle for eventually fully supporting Theia plugins and VS Code web extensions in the browser-only case, it will require some work.

Summary

With the support of browser-only applications we hope to open an exciting new chapter for the Theia framework. Please have a look at our PR and the hosted example applications and share your opinion below. Thank you!

[goyalyashpal]

so, this "backend-less/browser only" theia is to theia-desktop what github.dev or vscode.dev are to github.codespaces right?

[sdirix]

Yes that's a fair comparison when looking at the "github.dev web based editor". The clarification is needed as GitHub codespaces is also served via the github.dev domain.

[goyalyashpal]

this blog about vscode-dev lists many more (practical) scenarios where the serverless application becomes indispensible

my ~summary~ distillate of it: any scenario where full fledged desktop application can't be installed but a browser is available. so like:

• restricted machines: offices, schools, labs, libraries, etc
• restricted os-es: chromebook or window lite mode etc
• devices with unsupported os: e.g. mobile devices like android / ipad tablets etc

[msujew]

@goyalyashpal For these use cases, it's probably still best to have a full Theia instance running (for example in the cloud). The backend-less Theia version has other use-cases. For example, we're currently working on https://github.com/eclipse-theia/theia/issues/2842 which in conjunction with the backend-less Theia would allow users to join any workspace in the web without needing to setup a whole Theia server instance.

[erlmachinedev]

Extremely valuable contribution 🚀 I do Theia based project from the beginning and must admit that security considerations around backend-less option give me a way better confidence.

You have an option to control everything you want via dedicated service (the great way for virtualization of resources).

P.S. Taking into account extensible design of the tool I would wish (as a user) to see browser-only version first (where the backend comes as an option)