Can't import other user scripts in user scripts

[Claw76]

Plugin information:

• OS: Windows 10
• Templater version: 1.10.0
• Obsidian version: 0.13.23
• Templater settings: Template folder location: _templates/, Script folder location: _scripts/

Bug description: I import certain functions in my user scripts, which I also use as user scripts themselves. Due to the changes in Templater v1.10.0 this breaks and I get an error when importing other userscripts in a userscript. Works in Templater v1.9.11.

Example: Folder structure: _scripts:

• userscript1.js
• userscrtip2.js

userscript1.js:

function f1(msg) {
 console.log(msg)
}
module.exports=f1;

userscript2.js:

const f1 = require('./userscript1.js');
function f2(msg) {
    f1(msg);
}
module.exports = f2; 

The Error:

VM112:82 Templater Error: Template parsing error, aborting. Cannot find module './userscript1.js' Require stack: - electron/js2c/renderer_init

What I found out so far: Apparently, since Templater v1.10.0, the directory where the script is "located" when executed is now: C:\Users\Username\AppData\Local\Obsidian\resources\electron.asar\renderer when up until Templater v1.9.11 it was always the script-folder specified in the Templater options. Would be nice if that could be changed again.

Test Vault.zip

[nicolevanderhoeven]

Thank you for this @Claw76 . I've seen this error as well, and it's breaking some of my workflows. Would love for this to be fixed.

[cfurrow]

It seems that previously (in v1.9.11) the user function was parsed/loaded this way:

const vault_path = this.app.vault.adapter.getBasePath();
const file_path = `${vault_path}/${file.path}`;
//..snip
const user_function = await import(file_path);

That was changed in v1.10.0:

let req = (s: string) => {
    return window.require && window.require(s);
};
let exp: Record<string, unknown> = {};
let mod = {
    exports: exp
};

const file_content = await this.app.vault.read(file);
const wrapping_fn = window.eval("(function anonymous(require, module, exports){" + file_content + "\n})");
wrapping_fn(req, mod, exp);
const user_function = exp['default'] || mod.exports;

In v1.9.11, it seems Templater used the "native" await import to load the file, and put the contents into const user_function. That would probably have preserved the current working directory of that script file, and would have allowed loading of other files relative to it (e.g. ./user_function1.js, etc).

In v1.10.0, the user script file is not loaded via import, but instead, the contents of the script file are read into memory, and then window.eval is called on the contents. This presumably changes the entire context, and that's why you lose the ability to load files that are sitting in the same folder as your original file. For all intents and purposes, Templater has lost all knowledge and understanding of where the file used to be located on the filesystem, because all it cares about now is the contents, and it "pastes" that content into a window.eval.

I don't know how the current changes to Templater could be modified to allow relative file loading, like it had before, and I don't know the reasoning behind changing the behavior to what it is now (aside from attempting to do a little more code-sandboxing).

Maybe there's a happy-medium to be had, but I don't have all the information around the current changes.

[shabegom]

Great explanation @cfurrow!

This change was made to support loading JS user scripts on Obsidian mobile (which doesn't allow for dynamic imports in the previous version).

I think it is definitely worth looking into if we can change the directory eval is running in to support your use cases. This is going to be a tricky issue, so please be patient.

[shabegom]

I've started working on a possible solution on this branch:

https://github.com/SilentVoid13/Templater/tree/cwd-fix

It's a WIP, but would like some feedback if this addresses your use-cases.

Since the require is for another script file in the templater scripts folder, I'm basically inlining the required script by reading its contents and replacing require() with the function.

I'll probably need to make it recursive in case there is another require inside the script being inlined.

This way retains mobile compatibility.

Anything I'm not thinking of with this approach? Totally open to another way to fix this as long as scripts work on mobile.

[cfurrow]

Thanks for putting the time into starting that branch, @shabegom.

I think that will fix @Claw76 , @nicolevanderhoeven and @JeppeKlitgaard's issue.

There's an added benefit of the "inlining" bringing in all functions from your other scripts, not just the module.exports one.

/** ./date_utils.js */


function extractDate(str) {
  let dateMatch = /(\d{4})-(\d{2})-(\d{2})/.exec(str);
  if (!dateMatch) dateMatch = /(\d{4})(\d{2})(\d{2})/.exec(str);
  if (dateMatch) {
    let year = Number.parseInt(dateMatch[1]);
    let month = Number.parseInt(dateMatch[2]);
    let day = Number.parseInt(dateMatch[3]);
    return new Date(year, month-1, day);
  }

  return undefined;
}

// Returns true/false if the two dates are within 24hrs of each other
function datesWithin24Hours(date1, date2) {
  return Math.abs(date1.getTime() - date2.getTime()) <= 1000 * 60 * 60 * 24;
}

// You can still only export a single function
module.exports = extractDate;

You can use both functions defined in date_utils.js:

/** ./my_userscript.js */

const extractDate = require('./date_utils');

function myFunc() {
  const theDate = extractDate('2022-01-31');
  const otherDate = extractDate('2022-02-01');

  if( datesWithin24Hours(theDate, otherDate) ) {
    return "something useful";
  } else {
    return "something less useful";
  }
}

module.exports = myFunc

Source view from inside Obisidian's dev tools:

const extractDate = 
function extractDate(str) {
  //...snip...

  return undefined;
}

function datesWithin24Hours(date1, date2) {
  return Math.abs(date1.getTime() - date2.getTime()) <= 1000 * 60 * 60 * 24;
}

module.exports = extractDate;

async function myFunc() {
  //...snip...
}
//...snip...

[cfurrow]

@shabegom:

I'll probably need to make it recursive in case there is another require inside the script being inlined.

Good point. At that point, it may be worth making some inline-resolver class that can do that.

• Read the main script file into the "inline resolver" class
• resolver finds all "local" requires
• for each local require, read contents, pass to another new "inline resolver" class
• repeat forever

That could work, but I do wonder how much access to the node VM is available from within Obsidian. I've not written any plugins, so I don't know the API that's available. When you mentioned "worth looking into if we can change the directory eval is running in", that seems like the "cleaner" approach, but I have no idea what Obsidian's API exposes to plugins to allow changing the current directory, or execution context for window.eval, etc.

Anything I'm not thinking of with this approach? Totally open to another way to fix this as long as scripts work on mobile.

Your solution does "work" for the top-level require statements, as I demonstrated in my demo, but like you said, the main issue may be dealing with the recursive requires in all child scripts. That could get messy to resolve and then you're simply rewriting / mimicking the behaviors of the native require in some naive way. Maybe that's unavoidable.

[shabegom]

@cfurrow and company, I've opened a PR with my solution: #543

It would be great if you are able to pull down the PR and test this before I go ahead and merge. If that isn't possible, I'll look into making a beta release that can be installed via BRAT.

Changes since my last post:

• Is now a method that uses recursion to read over the file contents and replace require statements with the contents of the required file.
• looks for an existing function in the file before inlining. This prevents duplication of functions.
• trims the module.exports from the inlined contents (this didn't seem to be a problem, but better to be safe)

To your point, changing the execution context/working directory for window.eval is definitely a cleaner solution, but I couldn't find a way to do this and am worried it wouldn't work on the mobile app since directory structure is weird (especially on iOS). If anyone has a way that this could work, I'd happily take a PR with this approach.

One thing I think could be improved upon overall is how user functions are executing. When one user function is run, it is actually reading all of .js scripts in the script folder. I think a longer term solution would be to read these files into memory.

[cfurrow]

Fantastic, @shabegom ! Thank you again for donating your time to this issue. I'll try this out again when I get a chance. I ended up stumbling upon this github issue out of curiosity. I did not have an immediate need for the issue, but it's been interesting to follow along.

looks for an existing function in the file before inlining. This prevents duplication of functions.

Great idea. I forgot to mention that after I saw the functions object/hash--you get the "is this already loaded?" for free this way. Very cool.

couldn't find a way to do this and am worried it wouldn't work on the mobile app

Absolutely! My cursory investigation into this also yielded few results.

One thing I think could be improved upon overall is how user functions are executing. When one user function is run, it is actually reading all of .js scripts in the script folder. I think a longer term solution would be to read these files into memory.

Another good call out. I presume the plugin is re-reading all files all the time to collect updates on any file changes in the scripts folder, so going down this route may require updating the in-memory store of functions if changes are detected. Again, my inexperience with Obsidian's API has me wondering if it could be of any help here with events, or, even file MD5 hash comparisons of what is loaded in memory vs what is in the file. 🤷🏻

[JeppeKlitgaard]

@cfurrow @shabegom @SilentVoid13 Did this ever end up being solved? I just upgraded to 1.10+ and found that for me this is still an issue. Is there perhaps a new recommended way of accessing shared scripts?

[JeppeKlitgaard]

I've worked on migration over my functions to a new layout and I am happy to report that you are able to call 'tp.user.` functions from other user scripts now (I don't think this was always the case, or at least I couldn't get it to work back when this discussion was started).

This allows factoring out reusable bits into user scripts, which I think is sufficient that this issue can be closed now?

[bathrobe]

hey all thanks for this. just in case anyone who gets here later on is unsure how the nested user scripts should look, here is an example:

// scripts/testing2.js
const testing2 = () => {
  console.log("this is testing 2");
};

module.exports = testing2;

// scripts/testing.js
const testing = (tp) => {
  console.log("this is testing");
  tp.user.testing2();
};

module.exports = testing;

// ../templates/using_nested_scripts.md
<% tp.user.testing(tp)%>

// running using_nested_scripts.md:
// > this is testing
// > this is testing 2