[meta] Tracking issue for Node.js/Deno standard library inspired functions to include in `@tauri-apps/api`

[amrbashir]

I will be refactoring our js api and aiming to make the it more familiar to users coming from electron/js-land so it is easier to migrate.

Some of current functions will be deprecated and removed when new variants are released.

We want to gather some feedback from users and which functions they want to add from the Nodejs standard library, so comment below and I will add them to this list of new functions:

1. os module

   • [x] EOL
   • [x] platform()
   • [x] version()
   • [x] tempdir()
   • [ ] hostname

2. path module

   • [x] join()
   • [x] normalize()
   • [x] resolve()
   • [x] sep
   • [x] delimiter
   • [x] dirname()
   • [x] basename()
   • [x] extname()
   • [x] isAbsolute()

3. fs module

   • [ ] create()
   • [ ] open()
   • [ ] close()
   • [ ] FsFile class
   • [ ] writeFile()
   • [ ] writeTextFile()
   • [ ] readFile()
   • [ ] readTextFile()
   • [ ] copyFile()
   • [ ] readDir()
   • [ ] mkdir()
   • [ ] remove()
   • [ ] rename()
   • [ ] exists()

4. net module

   • [ ] createConnection()

5. dns module

[Shotman]

Suggestion : get complete directory tree as JSON from a given folder, along with files with their metadata getFileTree('dir')

[amrbashir]

Suggestion : get complete directory tree as JSON from a given folder, along with files with their metadata getFileTree('dir')

Sorry, I forgot to mention I am accepting only function suggestion from Nodejs standard library for now.

Once I am done with this, I think I will make a plugin for extended api and will accept suggestion for anything.

[Verequies]

It would be great to have 'net.createConnection' from the Net API so that discord-rpc can work. I would be happy to make an example project that can be tested with if needed.

[amrbashir]

It would be great to have 'net.createConnection' from the Net API so that discord-rpc can work.

I added createConnection to the list for now, but I will ask @nothingismagick , our security expert, if it is okay to include it or not since it will deal with sockets.

I would be happy to make an example project that can be tested with if needed.

I will try to do my best but I am not sure if this will be a drop in replacement for the NodeJS one.

[jvail]

Would be nice to have a synchronous api to access to files in a way that allows open/read/write/close (as mentions here https://github.com/tauri-apps/tauri/issues/1025) to directly read/write large files without copying them into memory:

Something like https://github.com/emscripten-core/emscripten/blob/main/src/library_nodefs.js

[amrbashir]

Would be nice to have a synchronous api to access to files in a way that allows open/read/write/close (as mentions here https://github.com/tauri-apps/tauri/issues/1025) to directly read/write large files without copying them into memory:

Something like https://github.com/emscripten-core/emscripten/blob/main/src/library_nodefs.js

A synchronous api is not possible, but I will add nodejs async function open which returns a FileHandle class

[jvail]

A synchronous api is not possible, but I will add nodejs async function open which returns a FileHandle class

Sorry, a bit off-topic but Is that a design decision or technically impossible?

I will be refactoring our js api and aiming to make the it more familiar to users coming from electron/js-land so it is easier to migrate

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

[amrbashir]

Sorry, a bit off-topic but Is that a design decision or technically impossible?

yeah it is a kinda impossible technically.

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

you can wrap your logic in an async function and await our api calls. here is an example using an async IIFE.

(async () => {
	doSomething();
	let file = await fs.readFile();
	doSomethingWithFile(file);
})();

[jquesada2016]

Sorry, a bit off-topic but Is that a design decision or technically impossible?

yeah it is a kinda impossible technically.

Since electron offers sync fs it will be difficult to migrate. Are there ways to customize/extend tauri to make a sync fs available in a webworker?

you can wrap your logic in an async function and await our api calls. here is an example using an async IIFE.

(async () => {
	doSomething();
	let file = await fs.readFile();
	doSomethingWithFile(file);
})();

Regarding sync function calls, I do understand the technical limitations on why promises must be used. But just in case, it is possible to emulate (though inefficiently) a blocking synchronous function in JS from an asynchronous function. This can be done as follows:

const syncFunctionNotDoneSymbol = Symbol(/* named, if you prefer */);
let valueToBeReturned = syncFunctionNotDoneSymbol;

export function syncFunction() {
    asyncFunctionCall().then(result => valueToBeReturned = result);

    while(valueToBeReturned === syncFunctionNotDoneSymbol) {
        // burn CPU cycles...
    }

    // Reset the value, so subsequent function calls still work.
    let val = valueToBeReturned;
    valueToBeReturned = syncFunctionNotDoneSymbol;

    return val;
}

Two important things to note on my primitive implementation above:

1. The functionality can be made into a simple helper function to convert any async function into a sync one (haven't looked in NPM to see if such a disgrace already exists...
2. I am not sure if my above implementation works if the function is called from worker threads, as I don't know if the symbol definition is re-evaluated when called from another thread context. Though it is definitely possible to make it work if it doesn't by default by using a simple array as a sort of concurrent queue for the symbols. If you would like me to show you an example, let me know.

It would also be a good idea, if such functionality is desired, to add a disclaimer in the sync function docs where you heavily discourage their use, due to them eating CPU cycles polling for the promise. But it might serve as a simple shim for large code bases until they are ported over.

[chippers]

Regarding sync function calls, I do understand the technical limitations on why promises must be used. But just in case, it is possible to emulate (though inefficiently) a blocking synchronous function in JS from an asynchronous function. This can be done as follows:

The impossible part isn't exposing a sync api, it was the other part of the request.

to directly read/write large files without copying them into memory

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed. To do that without copying into memory, there would need to be a way from JavaScript to directly load a file from the filesystem through the native WebView. Browsers don't have this, and AFAIK the native WebView libraries we use don't expose some way to do this.

As for your sync version of async methods, I'm not sure if we will expose pseudo sync functions. IMO it matters less to have them since all the supported browsers have await. If we don't expose them, it could still be a simple/contained package that wraps the async api. I also think that the JavaScript Generator API would more succinctly allow you to turn it into a sync call.

[jvail]

Thank you for your thoughts and insight, but ...

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed

excuse the heresy - this is pretty much what I can do already in a good old browser: Ask for a file, load it entirely into memory, process it and ask for a save-as dialog. This is fine for small files but let's say I'd like to process a 500MB geotiff then I'll soon hit the limitations of the browser (and tauri's).

The use-case I have in mind is running an emscripten compiled WebAssembly in a web worker to process local files. As of now I can not embed the wasm directly in rust (even if I could it has some advantages being able to run it within the webview). With electron I can run it in a worker and by setting nodeIntegrationInWorker: true the lib gets full read/write access to the filesystem.

I do not dare to ask for design changes but maybe there are ways to allow more customization on the rust level to inject an API (e.g. filesystem) in the worker context bypassing tauris async command (serde) api. Being able to combine WebAssembly, workers and tauri would create really interesting opportunities.

[chippers]

Thank you for your thoughts and insight, but ...

This is difficult in the current design because we use message passing to communicate with the WebView, meaning Rust loads the file and passes the bytes over to the webview into JS where it is consumed

excuse the heresy - this is pretty much what I can do already in a good old browser: Ask for a file, load it entirely into memory, process it and ask for a save-as dialog. This is fine for small files but let's say I'd like to process a 500MB geotiff then I'll soon hit the limitations of the browser (and tauri's).

The use-case I have in mind is running an emscripten compiled WebAssembly in a web worker to process local files. As of now I can not embed the wasm directly in rust (even if I could it has some advantages being able to run it within the webview). With electron I can run it in a worker and by setting nodeIntegrationInWorker: true the lib gets full read/write access to the filesystem.

I do not dare to ask for design changes but maybe there are ways to allow more customization on the rust level to inject an API (e.g. filesystem) in the worker context bypassing tauris async command (serde) api. Being able to combine WebAssembly, workers and tauri would create really interesting opportunities.

I agree with you, that would be a really interesting interface and I would love to see it someday soon. Electron is in a fortunate place regarding this due to controlling both the backend and a custom build of Chromium. We provide Wry by default in Tauri to support the "native" OS browser (iOS/macOS = WkWebView, Linux = webkit2gtk, Windows = webview2) which would require all those libraries to support such functionality, and then apply Tauri APIs over it.

As for regarding "this is pretty much what I can do already in a good old browser", that is relatively true. Browsers will let you open a local file from a prompt, some experimental APIs will even let you do it without a prompt, if the files are in the correct directory.

These types of things can likely be implemented by a Tauri plugin, but only if the underlying platform supports it. I haven't look into it at all, but if you are interested in it then the results may be fruitful.

Additionally, if you did want to support a custom runtime including things like preventing copying memory when reading large files, then you can implement Runtime from tauri-runtime as the backend of your Tauri application. It does take a decent chunk of work to implement your own runtime, so beware if you decide to do so.

[ghost]

I don't see it listed as a task (or maybe I misread), but adding file permissions when writing a file (not after the fact - race condition) would help tremendously for electron apps that needed to store sensitive data (e.g. passwords, secrets) that are not readable by groups or others.

Example:

// mode should be configurable, just showing as an example:
File::with_options().mode(600).open("somefile.txt");

[amrbashir]

@zoombinis it is listed under fs section, all current functions will have new overloads that is similar to nodejs standard library, with support for file permissions

Edit: updated the list for more transparency

[Lancear]

I really need the net.createConnection function for my app to create an IPC connection. How much work would it be to implement that? Could someone with little/basic Rust experience implement it? Would love to help with that if it means I can use Tauri instead of Electron for my app!

[amrbashir]

@Lancear we are in a code-freeze phase and there won't be any new features to Tauri v1, so any new features will land in Tauri v2 check #2060 for more info.

With that said, I gotta warn you, we might not be able to exactly replicate NodeJS' net.createConnection() but we will try our best. I also wanna note that fs module has higher priority than net module so it might be a while until we start on it.

Pull requests to the next branch are welcome though.

[JonasKruckenberg]

Just had a thought that maybe the dns module might be a good fit for this too as this can't be archived using webapis. Of course lower priority than os, path and fs but once we get around to the net stuff having dns would be cool too.

Spinning this further a plugin (core or not) could provide similar apis for mdns in the future

[amrbashir]

Just had a thought that maybe the dns module might be a good fit for this too as this can't be archived using webapis.

Sure let me add it to the list

Of course lower priority than os, path and fs but once we get around to the net stuff having dns would be cool too.

os and path are already done. fs is a different beast and won't add it in tauri v1 anyway so dns and net would also be planned for tauri v2

[lity]

The more NodeJS api to include the better, it's much easier and convenient for tauri users to write app logic in js layer than in rust layer.

[MaKleSoft]

Just adding my feedback here: We'd love to have access to os.hostname. Specifically, we'd be using it to generate more descriptive device names for things like listing active sessions, trusted devices etc.

[KaKi87]

Hello,

I would also like to request the ability to read a file line by line, like the following in Node :

import {
    createReadStream
} from 'node:fs';
import {
    createInterface
} from 'node:readline';
for await (const line of createInterface({
    input: createReadStream(
        path,
        {
            flags: 'a+'
        }
    ),
    crlfDelay: Infinity
})){
    // TODO something with `line`
}

Not necessarily by implementing exactly the same modules and methods as long as it does the same thing.

Thanks

[amrbashir]

@KaKi87 thanks for the suggestion, it has now been implemented in the linked PR and will look like this:

const lines = await readTextFileLines('file.txt', { baseDir: BaseDirectory.App });
for await (const line of lines) {
  console.log(line);
}

Also, since each line will be lazily read as you advance through the iterator, it makes this function perfect for performance critical apps as you can manually advance the iterator by calling .next() whenever you want.

[lucasfernog]

Nice one @amrbashir

[KaKi87]

each line will be lazily read as you advance through the iterator, it makes this function perfect for performance critical apps

Exactly, thanks !

[SamuelScheit]

Any updates on the net api?

[jlarmstrongiv]

Does the fs module support streaming apis like:

• fs.createReadStream
• fs.createWriteStream
• fs.readableWebStream

I was looking for ways to read files without loading the entire file into memory, but found issues like https://github.com/tauri-apps/tauri/issues/4133 and this one

Any workarounds for the current version of tauri?

[KaKi87]

Any workarounds for the current version of tauri?

If you need it that bad, then you could call an external tool (maybe cat) using shell.Command.

[jlarmstrongiv]

@KaKi87 that’s great for reading files! Do you have a workaround for writing files too?

[KaKi87]

echo.

[jlarmstrongiv]

Ahh, I was looking at the Command interface, not the Child interface. My bad. Thank you @KaKi87!