Parse from a file without copying to memory buffer?

[niklauslee]

Hello,

I want to use a filesystem (FATFS) in flash memory and run automatically index.js on boot. But I need to copy the entire content of the file to memory buffer to call jerry_parse().

Is there any way to parse and run code from read stream (or providing a read callback)?

Thanks,

[zherczeg]

No. But if your code is fixed, I recommend to use snapshots.

[niklauslee]

Thank you for your answer. I just want to share the context where this is necessary.

MicroPython and CircuitPython provides a filesystem (1MB in flash) as a USB drive when connect to the PC as shown in attached image. It runs code.py automatically on boot and support import modules from /lib directory. It parses the code from file read stream without allocating entire code in memory buffer.

I'm developing Node.js-like runtime for microcontrollers and want to introduce a small internal filesystem for code and modules. I would appreciate it if you could consider this function in the future.

Thank you.

[rerobika]

@niklauslee

I think static snapshot is what you are looking for. See: https://github.com/jerryscript-project/jerryscript/issues/1352#issuecomment-546495897.

[zherczeg]

JavaScript is way more complicated than python, reading the source in fragments would complicate the otherwise very complicated parser further. But you can skip the entire parsing process by using snapshots, which saves disk, memory, and runtime.

[niklauslee]

@rerobika @zherczeg I totally agree snapshot is enough to reducing disk and memory. But my problem is about convenience (rather than efficiency). Recent script runtimes for microcontrollers (MicroPython, CircuitPython, NodeMCU Lua) support internal file system, so developers can write code and save it to the filesystem directly using a popular code editor like VSCode, Sublime, etc. It really simplifies writing script codes for microcontrollers.

I'm also understand modifying parser reading source from stream is hard work. This issue could be closed if it is not necessary for jerryscript. Then, I should put user's js code directly in flash memory address area without filesystem so that jerryscript can parse and run directly with a pointer to the string of the code. (I am worried that snapshots will make debugging difficult)

Thank you for your advices.

[zherczeg]

What does "directly" means? I suspec there is some kind of tool to write the virtual file system, or you just put files into a real directory and pack them with a tool. In either case, you can run the snapshot generator automatically, so it should be convenient from user perspective (they don't even need to know that it is a snapshot). As a bonus you get early syntax checking.

I know the approach is somewhat different from others, but this is a matter of tooling, not convenience for me. Snapshots always superior to raw source code, I am surpreised the others cannot do this.

Regarding debugging, if you have enough memory for debugging, then parsing and throwing away the code should not be a problem as well.

[niklauslee]

@zherczeg You're right. Using a bundler (e.g. webpack) and snapshot generator, we can generate a compact snapshot. But my project is targeting beginners, students, coding newbies. I just wanted let them handles just source files only without using dedicated tools to generate snapshots. I think this is more natural for scripting languages :)