How should I load a module?

[ghost]

I wanted to implement data loaders in Javascript. As far as I understand, Duktape provides no I/O. So that's the first thing I have to implement, to run scripts from disk, and to load data from the script. However, some things are unclear. If I'll just read the whole script with, say, fread, and evaluate it somehow, how could I run a function from a script? I'd rather wanted to export a register function to the script. Then, from the C code, I would detect data type by trying registered functions, and then, would load the data using the detected appropriate script. So, in pseudocode:

loaders = []

load_data(path) {
    data = 0
    for loader in loaders
        data = loader(path)
        if data
            return data
    return data
}
real_main() {
    data = load_data("data.bin")
    do something with data
}
main() {
    create heap
    init duktape/c i/o
    run scripts that define loaders
    real_main()
    destroy heap
}

So the problem is - how can I iterate over Duktape functions in C code? If I don't even know will the functions be garbage collected after the script runs.

[svaarala]

There are two existing module loaders in the repo:

• https://github.com/svaarala/duktape/tree/master/extras/module-duktape
• https://github.com/svaarala/duktape/tree/master/extras/module-node

Both of these load modules by taking the module source code and encapsulating it in a function (...) { /*script here*/ } wrapper. The function is then called and it can e.g. register exported values via arguments provided to the wrapper function (e.g. an exports table).

There's no facility to load a script and enumerate functions inside it without actually running the script.

As for reachability, anything reachable from GC roots (including the value stack of reachable threads) will be reachable, even functions inside a compiled script etc.

[ghost]

https://stackoverflow.com/questions/36446481/in-duktape-how-to-convert-a-js-source-file-content-to-bytecode It says about duk_compile_lstring and duk_dump_function. What's the difference between them and modules, and what is preferred?

The function is then called and it can e.g. register exported values via arguments provided to the wrapper function (e.g. an exports table).

Can I have imports table? Say, a C function that takes a function as an argument? duk_push_c_function() seems to be right.

[svaarala]

It says about duk_compile_lstring and duk_dump_function. What's the difference between them and modules, and what is preferred?

When executing a script ("program") you first compile it, and then execute it. The result of compilation is an anonymous function which contains all the code (including inner functions) of the script. This is unrelated to modules as such.

Can I have imports table? Say, a C function that takes a function as an argument? duk_push_c_function() seems to be right.

I'm not sure what you mean - but you should probably look into the existing module loader examples to see how they work and whether they are appropriate in your use case.

At present Duktape itself does not understand or participate in module loading at all, so all the stuff related to how modules work are up to the externally implemented loader.

[ghost]

I'm not sure what you mean

I mean call C function from module with function as argument. I mean JS callback. In Python 3, that would be (I tested the code)

import numpy as np
loaders=[]

def register_loader(loader):# C function that I want to import to JS
    loaders.append(loader)
    return loader

def load(path): # that should be available to C
    for loader in loaders:
        data = loader(path)
        if data is not None:
            return data

@register_loader # register call, register_loader(my_loader). 
def my_loader(path): # that's the implementation, should be in JS
    def read_short(f):
        s = f.read(2)
        return s[0]|s[1]<<8
    with open(path, "rb") as f:
        if f.read(4) != b"FLAR":
            f.seek(-4, 1)
            return
        dim = ord(f.read(1))
        shape = []
        count = 1
        for _ in range(dim):
            sz = read_short(f)
            shape.append(sz)
            count *= sz
        return np.fromfile(f, dtype=np.float32, count=count).reshape(shape)

a = np.float32(np.random.randn(64,64))
with open("test.bin", "wb") as f:
    f.write(b"FLAR")
    f.write(bytes((len(a.shape),)))
    for sz in a.shape:
        f.write(bytes((sz&255, sz>>8,)))
    a.tofile(f)

b = load("test.bin")
should_be_true = np.allclose(a, b)
print(should_be_true)
assert(should_be_true)

Maybe it would be simpler to write everything in js, and provide a C load function that merely calls JS load function?