Mobile: Add support for plugins

[personalizedrefrigerator]

Summary

This pull request:

1. Adds a mobile-only default plugin that displays a message box
2. Loads and runs the plugin from a jpl file on mobile.
   • Plugins are run within a background WebView

Demo APK

Plugin settings GUI

What's working

• [x] Loading plugins from JPL files
• [x] Running plugins in a background WebView
• [x] joplin.views.dialogs.showMessageBox API
• [x] joplin.plugins.register API
• [x] Data API
  • Tested with getting a list of folders and adding a note to the first one.
• [x] CodeMirror 6 content scripts
  • Currently, the example "CodeMirror line numbers" plugin loads and adds line numbers to the editor
  • Some remaining issues (see below)
• [x] Dialogs
• [x] Markdown-it content scripts

To-do

• [x] Connect to the plugin repository
  • For now, this is only enabled on Android (and iOS in dev mode)
  • To enable this on iOS in production, extra work may be required to comply with iOS AppStore guidelines. For example, we might use a plugin repository in which all plugins are pre-reviewed by Joplin team members. Additionally, such plugins may need additional restrictions to comply with the requirement that code not included in the bundle "only use capabilities available in a standard WebKit view"^1 which may limit what APIs such plugins can use.
  • [x] Verify that updates work correctly
    • Checked & some bugs were fixed — it will need to be tested again.
• [x] Allow loading plugins from .jpl files
  • Currently disabled on iOS release builds (enabled in development builds) for reasons mentioned above.
• [x] Fix plugins don't reload correctly with React Native's fast refresh.
  • [ ] Follow-up: Implement PluginRunner.stop for desktop?
• [x] Add GUI for enabling/disabling plugins
  • Currently, this uses PluginRunner.stop when a plugin is disabled (rather than requiring an app restart). This reuses the logic that adds support for React Native's fast refresh.
• [x] Test on iOS
• [ ] Get plugin API working
  • [x] CodeMirror 6 content scripts
    • [x] Load content scripts into editor
    • [x] postMessage
    • [x] Fix content scripts not added if the plugin finishes loading while the editor is open. (Editor needs to be closed & re-opened).
  • [x] Toolbar buttons
  • [x] WebView dialogs
  • [x] Panels (e.g. as used by the favorites plugin).
    • We also need some way to move/resize/switch between panels. As this could get complicated, it may be best left for a follow-up pull request.
  • [x] Show plugin error messages in the log and link to them in settings.
  • [ ] Imaging API?
  • [ ] Clipboard API
    • [x] Text
    • [ ] HTML
      • Doesn't seem to be supported by the clipboard library we use
  • [x] Renderer content scripts
    • [x] Run renderer within the note viewer iframe (DOM API access & allows us to avoid evaling plugin code in the main process).
      • [ ] Check performance: How much slower (if at all?) is it?
    • [x] Support webviewApi.postMessage
• [x] Fix error messages when there are multiple plugins registered
• [ ] Document plugin development workflow
  • [ ] Will plugin authors need to run a development version of Joplin to load & develop plugins?
• [ ] Move refactoring changes to separate pull requests. These changes are currently part of this pull request, but should be moved to separate pull requests.
  • [x] Mobile: Markdown toolbar refactor
  • [ ] Mobile: CodeMirror: Moved padding from the outside of the editor to the inside.
  • [ ] @joplin/default-plugins: Added support for building for just one application and refactored default plugins configuration file from JSON to TypeScript.

Notes

• The expo-asset library is used to load local assets (used to load JPL files)
• Plugins are run within iframes within a single WebView
• RemoteMessenger and tarExtract were originally created for other pull requests
• We work around this buffer bug to support tarExtract
  • This might instead be a React Native Hermes bug? (Upgrading RN might help.)
• Whether or not a plugin is running on mobile can be found through the versionInfo().platform plugin API.

Code structure

The below explanation(s) may be converted into files under readme/dev/spec.

Explanation of RemoteMessenger

On mobile, not only is the background page running in a separate process, but so are the renderer and dialogs.

To simplify communication between these processes, a RemoteMessenger class is introduced.

The RemoteMessenger<LocalInterface, RemoteInterface> class

The RemoteMessenger class simplifies communication over postMessage. Its job is to convert asynchronous method calls to messages, then send these messages to another RemoteMessenger that handles them.

flowchart
	RemoteMessenger1<--postMessage-->RemoteMessenger2

For example, if we have

// Dialogs
export interface MainProcApi {
	onSubmit: ()=> void;
	onDismiss: ()=> void;
	onError: (message: string)=> Promise<void>;
}

export interface WebViewApi {
	setCss: (css: string)=> void;
	closeDialog: ()=> Promise<void>;
	setButtons: (buttons: ButtonSpec[])=> void;
}

We might then create messengers like this:

In the WebView:

const webViewApiImpl: WebViewApi = {
	// ... some implementation here ...
	setCss: css => {} // ...
};

// Different messageChannelIds allow us to have multiple messengers communicate over the same channel.
// Different IDs prevent the wrong messenger from acting on a message.
const messageChannelId = 'test-channel';

const messenger = new WebViewToRNMessenger<WebViewApi, MainProcApi>(messageChannelId, webViewApiImpl);

In the main process:

const mainProcApiImpl: WebViewApi = {
	// ... some implementation here ...
	closeDialog: () => {} // ...
};

const messageChannelId = 'test-channel';
const messenger = new WebViewToRNMessenger<MainProcApi, WebViewApi>(messageChannelId, mainProcApiImpl);

// We can now use the messenger.
// Messages are all asynchronous.
await messenger.remoteApi.setCss('* { color: red; }');

To call messenger.remoteApi.setCss(...), we use a process similar to the following:

First: Queue the method call and wait for both messengers to be ready.

When a messenger is ready, it sends a message with kind: RemoteReady.

flowchart
	postMessage1(["postMessage({ kind: RemoteReady, ... })"])
	rm1--1-->postMessage1--2-->rm2
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
	end

When a messenger receives a message with kind: RemoteReady, it replies with the same message type.

flowchart
	postMessage1(["postMessage({ kind: RemoteReady, ... })"])
	rm2--3-->postMessage1--4-->rm1
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
	end

Second: Send all queued messages

After both messengers are ready, we wend all queued messages. In this case, that's the setCss message:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['setCss'],
	arguments: {
		serializable: ['* { color: red; }'],

		// If there were callbacks, we would assign them
		// IDs and send the IDs here.
		callbacks: [ null ],
	},
}

flowchart
	postMessage(["postMessage({ kind: InvokeMethod, ... })"])
	rm1--2-->postMessage--3-->rm2
	subgraph MainProcess
		call(["await m1.remoteApi.setCss('...')"])
		call--1-->rm1
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
	end
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
		webViewApiImpl["webViewApiImpl.setCss"]
		rm2--4-->webViewApiImpl
	end

After handling the message, a result is returned also by postMessage, this time with the kind ReturnValueResponse:

flowchart
	postMessage(["postMessage({ kind: ReturnValueResponse, ... })"])
	rm2--6-->postMessage--7-->rm1
	subgraph WebView
		rm2["RemoteMessenger< WebViewApi,MainProcApi >"]
		webViewApiImpl["webViewApiImpl.setCss"]
		webViewApiImpl--5-->rm2
	end
	subgraph MainProcess
		rm1["m1 = RemoteMessenger< MainProcApi,WebViewApi >"]
		calll(["await m1.remoteApi.setCss('...')"])
		rm1--8-->calll
	end

After receiving the response, the setCss call resolves.

On mobile, we address the same problem in similar, but more generalized way. We define a RemoteMessenger class that handles postMessage communication.

RemoteMessenger and callbacks

Suppose we call a method in a way similar to the following:

messenger.remoteApi.joplin.plugins.register({
	onStart: async () => {
		console.log('testing');
	},
	test: 'test',
});

We can't send callbacks over postMessage. As such, we assign the onStart callback an ID and send the ID instead. The message might look like this:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['joplin', 'plugins', 'register'],
	arguments: {
		serializable: [
			{
				onStart: null,
				test: 'test',
			}
		],
		callbacks: [
			{
				onStart: 'some-generated-id-for-onStart',
				test: null,
			}
		],
	},
	respondWithId: 'another-autogenerated-id',
}

Note: As before, the respondWithId connects a method call to its return value (the return value has the same ID).

The arguments.callbacks object contains only callback IDs and the arguments.serializable object contains only the serializable arguments. The two objects otherwise should have the same structure. These two objects are merged by the RemoteMessenger that receives the message:

flowchart
	callbacks[arguments.callbacks]
	serializable[arguments.serializable]

	callbacks--"only callbacks"-->original
	serializable--"only properties not in callbacks"-->original

Callbacks are called by sending an InvokeMethod message similar to the following:

{
	kind: MessageType.InvokeMethod,
	methodPath: ['__callbacks', 'callback-id-here'],
	arguments: { ... },
	respondWithId: 'some-autogenerated-id-here',
}

[personalizedrefrigerator]

I'm marking this as ready for an initial review.

While there is still work left to be done (plugin panels, renderer performance improvements, code quality, and tests where possible), the core functionality and code structure should be present.

The following pull requests include changes backported from this pull request. Reviewing and merging them should make reviewing this pull request more manageable:

• https://github.com/laurent22/joplin/pull/9708
• https://github.com/laurent22/joplin/pull/9728
• https://github.com/laurent22/joplin/pull/9726
• https://github.com/laurent22/joplin/pull/9701

[personalizedrefrigerator]

I'm marking this as a draft again. While testing with the RevealJS slides plugin, I found a few issues:

• Scripts aren't loaded correctly in dialogs.
• Large plugins are very slow to load.
  • Currently, we re-extract plugins from .jpl files each time they're loaded. This is very slow on mobile in part because binary file access is done through base64 conversions. Ideally, if a plugin hasn't changed, we shouldn't re-extract the plugin.
  • Edit: We don't re-extract .jpl files on each load. The slowness is likely from .injectJS or md5File.
  • Edit 2: md5File's current implementation is very slow (e.g. 6 seconds for a few megabytes).
• Large content scripts are very slow to load.
  • Currently, content scripts are loaded by 1) reading the content script file and 2) sending them to the WebView through injectJs. This is very slow for large content scripts (i.e. a few megabytes). We should instead copy the content script files to a directory readable by the WebView and include them with <script> tags. We can perhaps compare file modification times to determine whether we should re-copy a script or not.

[laurent22]

@personalizedrefrigerator, please let me know when this is ready to merge (there are few conflicts and the PR is still a draft)

[personalizedrefrigerator]

Closing -- all functionality, except default plugins, has been moved into separate pull requests.