[WIP] [Draft] Upgrade to Deno 1.30.3

[DavidDeSimone]

This PR is a work in progress (WIP) PR to upgrade to Deno 1.30.3. This PR will attempt to maintain backwards compatibility while adding new APIs.

This PR changes the way we implement deno. Previously, deno was run on the same thread as the lisp VM. This enabled a high degree of interoperability, but cause some performance issues. Specifically, we had to manually pump the deno event loop via a lisp timer - a process that was very fragile. Certain deno operations will only do so much work within a single cycle of the event loop. You wouldn't notice this in deno mainline because deno normally has full control over it's own event loop. Taking that control from deno and putting it under the lisp VM has a large impact on our ability to async load files.

Now, we will have 3 threads:

1. The main lisp thread
2. The deno thread
3. A lisp worker thread

When deno wishes to execute a lisp function, it will be querying a lisp worker thread. This allows us to solve a problem that ng-js had in calling lisp -> js -> lisp where the main thread is waiting on the js thread, and the js thread is waiting on the main thread.

This will make our new APIs all async/future based. My goal will be that calling

(js-eval-string-async "...") will return a 'future' that you can get the result of via (js-resolve future)

Our previously sync functions will just automatically resolve the future for you - keeping old behavior.

On the JS side, the lisp.function style functions now become async and will require

await lisp.function().

Goals

My goals are to improve performance, and to set up the stage to be able to use this code across different platforms. emacs-ng JS functionality still cannot be used on windows - my long term solution for that is to allow this code to also be used as an emacs module, and to allow dynamic linking on windows to solve that issue.

Remaining Work (will be updated over time)

• [x] Finish hooking up lisp worker -> js flow
• [x] Adding 'future' functionality for non-blocking JS tasks
• [x] Add JS/TS region/buffer functions
• [ ] Handle lisp proxies ~~- Add lisp function proxies (May be cut due to difficulty)~~

[DavidDeSimone]

I'm anticipating about 1-2 weeks from today (18FEB) until I have this in a polished usable state - from there I can build on top of it and iterate further. Currently, you can evaluate javascript and get the result to lisp, and you can call lisp -> js -> lisp. I need a better degree of auto marshalling, along with stripping color console characters from the deno output.

[DavidDeSimone]

Making good progress - the last three major items are:

1. Lower the download size (maybe pre-compiled, maybe strip the stuff we aren't using out of deno).
2. Proxies
3. Better error handling around the lisp -> js -> lisp flow.

[DavidDeSimone]

This patch is currently in a usable state - you can call JS from lisp, and query lisp from JS. The biggest remaining items are better error handling around JS (lips errors throwing a JS error when the promise resolves), and lisp proxies.

Proxies are what allows code like this:

const proxy = await lisp.currentBuffer();
const result = await lisp.bufferSize(proxy);

In that context, proxy is a special object that represents a lisp object.

Now, I do have eval'ing lisp from js working, so it may be that proxies are actually just increasing the complexity here more then they are worth. Instead, I could focus on an API that manipulates the world of the editor via eval or primitive calls.

[declantsien]

David DeSimone @.***> writes:

@DavidDeSimone commented on this pull request.

@@ -14,13 +14,18 @@ lsp_json = { version = "0.1.0", path = "../lsp_json" } libc = "0.2.95" lazy_static = "1.2" futures = "0.3" +deno = { git = "https://github.com/DavidDeSimone/deno", branch = @.***" }

@declantsien I made some changes for the deno submodule to reduce clone size - can you let me know if it is still taking a long time for initial clone for you?

-- Reply to this email directly or view it on GitHub: https://github.com/emacs-ng/emacs-ng/pull/463#discussion_r1130669744 You are receiving this because you were mentioned.

Message ID: @.***>

Yeah, the download time is way fast now. But I am getting this error

ld: alloc.o: in function garbage_collect': /shared/src/WREmacs/src/alloc.c:6063: undefined reference to Fjs__sweep' collect2: error: ld returned 1 exit status

[DavidDeSimone]

David DeSimone @.> writes: @DavidDeSimone commented on this pull request. > @@ -14,13 +14,18 @@ lsp_json = { version = "0.1.0", path = "../lsp_json" } libc = "0.2.95" lazy_static = "1.2" futures = "0.3" +deno = { git = "https://github.com/DavidDeSimone/deno", branch = @." } @declantsien I made some changes for the deno submodule to reduce clone size - can you let me know if it is still taking a long time for initial clone for you? -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: @.***> Yeah, the download time is way fast now. But I am getting this error ld: alloc.o: in function garbage_collect': /shared/src/WREmacs/src/alloc.c:6063: undefined reference to Fjs__sweep' collect2: error: ld returned 1 exit status

My bad, I was a little too aggressive in my deletions, should be fixed now.

[declantsien]

David DeSimone @.***> writes:

David DeSimone @.***> writes:

My bad, I was a little too aggressive in my deletions, should be fixed now.

-- Reply to this email directly or view it on GitHub: https://github.com/emacs-ng/emacs-ng/pull/463#issuecomment-1461645241 You are receiving this because you were mentioned.

Message ID: @.***>

Well, it compiles and runs now. But I have no idea how to use it in Emacs. Need sometime to figure that now. Can't remember how this used to work.

const proxy = await lisp.currentBuffer();
const result = await lisp.bufferSize(proxy);
console.log(result);

Where is the console.log output?

Also I tested some typo, there is no error popup with js-eval-buffer

const proxy = await lisp.currentBuffer();
const result = await lisp.bufferSize(proxy);
consoledsf.log(result);

I guess I am using it the wrong way. Does the js documentation needs updates?

[DavidDeSimone]

The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like

(js-resolve (js-eval-string "console.log('hello js');"))

There is also (js-eval-file "filepath") for an entire JS file.

Once in JS, you call lisp via the special lisp object. It auto-magically translates from camelCase to kebab-case.

await lisp.print("hello");

console.log should log to term if you launch the GUI through term.

[declantsien]

David DeSimone @.***> writes:

The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like

(js-resolve (js-eval-string "console.log('hello js');")

console.log should log to term if you launch the GUI through term.

-- Reply to this email directly or view it on GitHub: https://github.com/emacs-ng/emacs-ng/pull/463#issuecomment-1461699759 You are receiving this because you were mentioned.

Message ID: @.***>

Yeah, I saw some output. Not sure whether it is from js.

Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004

For this (js-resolve (js-eval-string "console.log('hello js');"), I am have Emacs backtrace popuo now.

Debugger entered--Lisp error: (tty text nil) js-resolve(nil) (progn (js-resolve (js-eval-string "console.log('hello js');"))) eval((progn (js-resolve (js-eval-string "console.log('hello js');"))) t)

Nice work BTW.

[DavidDeSimone]

David DeSimone @.> writes: The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like (js-resolve (js-eval-string "console.log('hello js');") console.log should log to term if you launch the GUI through term. -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: @.> Yeah, I saw some output. Not sure whether it is from js. Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 For this (js-resolve (js-eval-string "console.log('hello js');"), I am have Emacs backtrace popuo now. Debugger entered--Lisp error: (tty text nil) js-resolve(nil) (progn (js-resolve (js-eval-string "console.log('hello js');"))) eval((progn (js-resolve (js-eval-string "console.log('hello js');"))) t) Nice work BTW.

I also forgot - you need to call (js-initialize) first to start the deno env. That won't be required in the final API, but is just part of my PR for now. though I haven't seen those errors you were seeing, they may or may not be related.

[declantsien]

David DeSimone @.***> writes:

I also forgot - you need to call (js-initialize) first to start the deno env. That won't be in the final API, but is just part of my PR for now. though I haven't seen those errors you were seeing, they may or may not be related.

-- Reply to this email directly or view it on GitHub: https://github.com/emacs-ng/emacs-ng/pull/463#issuecomment-1462570009 You are receiving this because you were mentioned.

Message ID: @.***>

I guess I just did thing the wrong way then. Will wait for the final API got stabilized.

For your information (js-initialize) gives me this.

Fatal error 6: Aborted Aborted

[appetrosyan]

Any chance we can help out with this?

[DavidDeSimone]

Any chance we can help out with this?

Hello,

I got a little distracted and never ended up finishing this, but it is in a pretty close to complete state. The biggest thing left was proxies, i.e. having a handle between JS objects and lisp objects. JS objects can have a hidden "value" field that isn't accessible directly by JS, but only the bindings. Our initial implementation stuck the lisp JS pointer as that data - though I think for this implementation having a "handle" (a hashmap key) into a map would likely be a better way, though the downside is that is requires a lookup each iteration isn't of just accessing the lisp object.

[DavidDeSimone]

Update on this - I think that pulling the JS VM into a separate thread ended up being a little too ambitious. I am going to try a new approach where I just upgrade deno to latest using the old VM/Sub-VM approach to get JS working again - from there, I think I can write a similar concept of the off-thread VM at the scripting layer.