tp.file.move() creates duplicates

[frankreporting]

Plugin informations (please complete the following information):

• OS: macOS Big Sur
• Templater version: 1.8.1
• Obsidian version: 0.12.13
• Templater settings: Template folder location: "00_Meta/01_Templates/", Timeout: "5"

Describe the bug I have an empty note template checking the file title, and based on that, including another template. After trying multiple approaches, I'm pretty consistently seeing either the included template does NOT render, or else it DOES render, but I end up with duplicate files in different folders (one where I moved from, and one where I moved to), or else I get duplicate files AND the included template fails to render.

Expected behavior In empty note, using Empty Note template, if file title matches daily note format, i.e. YYYY-MM-DD, then include Daily Note template.

Additional context Error message in the console: "Template parsing error, aborting. Destination file already exists!" This is a confusing error, because my empty files are created in the current folder and I'm not creating the file from within the daily notes folder (10_Dated/11_Journal). When I log the tp.file.path() I get the folder where the note is created (40_Projects), so I'm unclear how the file could already exist in the destination 11_Journal. The only hint is that when I was logging info from both the Empty Note template and the Daily Note template, it appeared that tp.file.move() was being called TWICE, even though it's only coded into ONE of the templates. The first move is successful, but the original file appears to still exist, and it's almost like it's then trying to loop back and run the Empty Note template again, which is including the Daily Note again, which is trying to tp.file.move() to the daily notes folder, again. Since it seemed to have already created the file on the first round, it's throwing an error that the destination file already exists. I just can't figure out what could possibly be causing this loop-like behavior.

I've seen similar results whether the tp.file.move() is in the Empty Note template or in the Daily Note template.

Code Samples

Empty Note template:

<%* if (tp.file.title.search(/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) >= 0) {
await tp.file.include('[[Daily Note]]');
if (tp.file.path != "/10_Dated/11_Journal") { 
await tp.file.move("/10_Dated/11_Journal/" + tp.file.title);
}} else { %>---
id: <% tp.date.now("YYYYMMDDHHmmssSSS") %>
---
<%* } %>

Daily Note template:

---
alias: <% tp.date.now("M/D/YY dddd", 0, tp.file.title, "YYYY-MM-DD") %>
---
# <% tp.date.now("dddd, MMMM DD, YYYY", 0, tp.file.title, "YYYY-MM-DD") %>

[welpdx]

I do not think there is anything wrong with the Templater plugin here. I do notice some things regarding your script:

1. tp.file.title.search() returns either -1 (if not matching) and 0 (if match). Therefore, I changed that line to if (tp.file.title.search(/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) !=-1)

2. I've never used tp.file.include(). But during my testing, await tp.file.include() does not output anything. Not sure why. but doing something like tR += await tp.file.include("[[Daily Note]]") will for sure output the contents of [[Daily Note]].

3. await tp.file.move("/10_Dated/11_Journal/" + tp.file.title); worked fine for me.

4. I think the else statement

else { %>---
id: <% tp.date.now("YYYYMMDDHHmmssSSS") %>
---
<%* } 

should be placed along side the first if statement, not the nested if statement. I assume that you want to add that metadata if the file name is not a daily note file. Because your current script is saying:

if the note title is a daily note file 
     ...
     if the path is in "/10_Dated/11_Journal/"
     ...
     else
          add the meta data

I assume you want

if the note title is a daily note file 
     ...
     if the path is in "/10_Dated/11_Journal/"
     ...
else
     add the meta data

This misplaced else statement might be the reason why you thought that Templater is not executing your script.

5. I generally dislike putting <%* and %> everywhere. Instead, I just do tR +=

   for a cleaner result. Therefore, for your last else statement, I added what you wanted in a single line: tR += "---\n" + "id: " + tp.date.now("YYYYMMDDHHmmssSSS") + "\n---"

6. When testing the updated script, I do not notice any duplicates.

---

So, for your Empty Note Template, try this script:

<%* 
if (tp.file.title.search(/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) !=-1)
{
	let dailyNoteContent = await tp.file.include("[[Daily Note]]") 
	tR+= dailyNoteContent;
	if (tp.file.path != "/10_Dated/11_Journal")
	{
		await tp.file.move("/10_Dated/11_Journal/" + tp.file.title);
	}
}else
	{
		tR += "---\n" + "id: " + tp.date.now("YYYYMMDDHHmmssSSS") + "\n---";
	}
%>

I think it would now do what you expected it to do. Hope that helps.

[frankreporting]

hmmm.. First, thank you @welpdx for your detailed response. Very helpful!

At this point, though, I'm questioning my sanity. I put your code in place and it seemed to work fine for a few files. But after testing repeatedly, I eventually get the same error: Destination file already exists! And then it seems inconsistent. I can't always reproduce it. Screencap shows what I'm talking about:

[welpdx]

Ok I just realized several things:

1. path in tp.file.path is not a string like in Obsidian API's Tfile class path, or tp.file.title. According to Templater API, it is actually tp.file.path() where path() is a function. Also, to make the output of the path() function to output a relative, we need to do tp.file.path(true).

2. Also, in case you wanted to get the non-relative path, we can also use the includes(string) function to check if a specific substring occurs in string. I.e. since tp.file.path() gives us C:\Obsidian\vault1\foo\bar, tp.file.path().includes("foo\bar") would still return true because it does include that "foo\bar" substring.

3. If I manually create duplicate files (because Obsidian wont let me create a duplicate file in the app), and I run the script on the file that is not in the destination folder, I do get your error. I have to remove the duplicate first.

Here's the update script point 1 and 2:

<%* 
var desiredPath = "10_Dated/11_Journal"
if (tp.file.title.search(/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/) !=-1)
{
	let dailyNoteContent = await tp.file.include("[[Daily Note]]") 
	tR+= dailyNoteContent;
	if (tp.file.path(true).includes(desiredPath) != true)
	{
		await tp.file.move("/10_Dated/11_Journal/" + tp.file.title);
		new Notice("Moved \"" + tp.file.title + "\" to: " + desiredPath)
	}
} else
	{
		tR += "---\n" + "id: " + tp.date.now("YYYYMMDDHHmmssSSS") + "\n---";
	}
%>

Let me know if that works.

Edit: I added 2 minor adjustments:

• new variable desiredPath at the start to avoid hard coding.
• the script makes a popup toast (new Notice()) after it runs the tp.file.move() line. It is also useful for diagnosis. Remove it if it is annoying

[frankreporting]

Same issue.. 🤷 😳 😬

I will say the errors seem to start after I've successfully run it once or twice, then deleted the newly created files, and then run it again. I thought for a while maybe it had something to do with the way Obsidian stores information about the file system in the vault, but it's not ONLY happening on files that have at least been created once before. Once I started seeing the errors again, I tried it on files I've never created before, and it's now consistently failing again (destination file already exists, and I end up with copies in the source and target directories). It really seems like a bug.

Edit: That said, maybe it's an issue with tp.file.include() and not tp.file.move(). I have other templates that move files without any issue. It's only this one, with a combination include/move, that fails.

[welpdx]

Dang that sucks, @frankreporting. Can you give me a demonstration?

• I know that Obsidian caches scripts. Would Ctrl + R once in a while to reload Obsidian help? Based on the gist of you are saying, reloading it may help.
• And right now the script is running flawlessly, with a [[Daily Note]] that only has text. Do you have other scripts in [[Daily Note]]?
• By the way, are there scripts running automatically as you create new files? In the gif demo you linked above, it seemed like there is some automation going on. Idk how that works but it might be that that could mess things up.

Edit: I've added the script as an Empty Note Template, and I've been creating and deleting notes with valid and invalid year titles. Seems fine to me... Please see gif. I suspect it is not Obsidian cache but something else. Does including [[Daily Notes]] trigger other scripts? Also, are you using both Templater and Obsidian Templates?

[frankreporting]

Core Templates plugin is disabled, but I do have Templater and Dataview scripts inside the [[Daily Note]]. No other scripts run when Empty Note/Daily Note are created (I'm not entirely sure how I would set up an automation like that). Here's the full content of the Daily Note template (closing ticks for dataview codeblocks removed bc I couldn't figure out how to escape them properly in a Github message):

---
alias: <% tp.date.now("M/D/YY dddd", 0, tp.file.title, "YYYY-MM-DD") %>
---
# <% tp.date.now("dddd, MMMM DD, YYYY", 0, tp.file.title, "YYYY-MM-DD") %>



### Work Notes


# Overview

## To Do

```dataview
list due.file.day from #task
where !completed.file and start.file and start.file.day <= this.file.day and (!due.file or due.file.day >= this.file.day)
sort file.ctime desc

## Overdue

```dataview
list due.file.day from #task 
where !completed.file and due.file and due.file.day < this.file.day
sort file.ctime desc

## Deadlines Nearing

```dataview
list due.file.day from #task 
where !completed.file and due.file and due.file.day < (this.file.day + dur(7 days)) and due.file.day >= this.file.day
sort due.file.day asc

## Active Projects

```dataview
list due.file.day from #project 
where !completed.file and start.file and start.file.day <= this.file.day
sort due.file.day asc

## Events

```dataview
list
where file.name != this.file.name and file.day = this.file.day
sort file.ctime asc

## Created

```dataview
list
where file.name != this.file.name and file.ctime > this.file.day and file.ctime < (this.file.day + dur(1 day)) and file.day != this.file.day
sort file.ctime asc

[frankreporting]

And here's what happens when I set it up like you did in your screencap...

[welpdx]

This is strange. It might be something to do with tp then. You might be right.

I will make a script tmr with using only Obsidian API functions and see if that would work.

[welpdx]

Hey, I'm looking into this once again, and I'm thinking that it has something to do with your setup. When I trying the setup on a test vault, it all seems fine. gif Does the error still show for you?

As I am watching your gif, I notice that the error pops up after the debug toast. And sure enough, the newly created note seems to disappear in the __TEST folder but shortly returning. It seemed like move() worked but an error occured short after. Perhaps a duplicate is created? Perhaps another script was triggered upon new file creation?

Can you let me know if:

1. You see a duplicate file in destination folder?
2. The default Template plugin is off?

Welp

[mymykyta]

Heya, everyone. I've come across this exact problem myself, and at least on my system it seems to do with Templater. I'd spent loads of time trying to find a mistake in my syntax, but after stripping the 'new note' template to a single simple command, it still fails - what's worse is that it fails only sometimes, which (to me) points to some kind of internal issue.

I'm doing this on a Windows 10 system. Honestly, if it's not a Templater thing, I'm not sure what the issue could be. Another culprits could be my OneDrive or Obsidian Sync (?) maybe they are indexing or uploading files in the background which disrupts Templater from moving (no clue, and hopefully it's not them) - will look into those later.

Would love to find a fix for this, because auto-filing notes into folders is a huge workflow tool for me.

[mymykyta]

I'm doing this on a Windows 10 system. Honestly, if it's not a Templater thing, I'm not sure what the issue could be. Another culprits could be my OneDrive or Obsidian Sync (?) maybe they are indexing or uploading files in the background which disrupts Templater from moving (no clue, and hopefully it's not them) - will look into those later.

Okay, so after looking into it, and turning off all plugins, I've found that it's definitely a compatibility issue with Obsidian Sync. As soon as I even pause the syncing, the tp.file.move(...) works perfectly!

Would there be a way to alleviate this desync to keep both the Templater and Obsidian Sync running with some Templater JS magic, or is this looking like a more serious issue? I've mucked around a bit, trying to use an async function + delay() to create maybe some kind of time buffer, but I don't know javascript, so it's not working for me 🤷‍♀️

[welpdx]

Thanks @mymykyta for finding out that it was something to do with Obsidian Sync. I don't use Sync so I guess I never encountered that problem. (I use Obsidian-Git).

Yea when I say your first post, I thought about suggesting adding a pause in the script. I have this code below that I use to do that. Maybe add wait(1000) after the await tp.file.includes() function?

...
// this Wait function adds a 1 second pause 
function wait(ms){
   var start = new Date().getTime();
   var end = start;
   while(end < start + ms) {
     end = new Date().getTime();
  }
}
wait(1000);
new Notice("1 sec is up",1000)

[mymykyta]

Hi, thanks for your suggestion @welpdx. I tried running the script below on new file creation, but it's not fixed the issue. I have new insight though.

<%*
// this Wait function adds a 2 second pause 
function wait(ms){
   var start = new Date().getTime();
   var end = start;
   while(end < start + ms) {
     end = new Date().getTime();
  }
}
wait(2000);
new Notice("2 secs is up",2000)
await tp.file.move("50 Slipbox/" + tp.file.title)
%>

From the gif you can see what happens when the script is ran:

1. File is created in standard location.
2. The wait() function waits for 2 seconds.
3. File is moved into the new location, and a duplicate is created in the original location.
4. The duplicate starts running the templater script, counting to 2 sec with wait().
5. After the duplicate timer runs out, templater tries to run tp.file.move(), but because there is already a file in that place it gives the error:

Template parsing error, aborting. Destination file already exists!

So the added delay doesn't fix it, but it is helpful to understand the order of operations.

Potential Hack?

After tring a few more options I found that these two are the best, because they both result in destination files having run the Templater scripts (tho they still give errors).

• <%tp.file.move("<destination>" + tp.file.title)%>
• <%*await tp.file.move("<destination>" + tp.file.title)%>

We can use one of these to move the file to new place. Then the script would need to check if destination folder already has a file with the same file title. If it doesn't, we move the file into the desired folder. If one exists, then you just delete the current running one. This would basically, make the duplicates that are left behind trash themselves, since they still run the command (hence the errors).

I think this hack should not result in any data loss. If you run it only in newly created files, it will always prioritise keeping the existing ones over newly added ones... If you know of a better alternative workaround or a proper fix, I'd gladly try it, because atm I don't know if deleting is even possible with Templater, or how I would go about doing it.

PS: The first time, when I ran the script above for the gif, it actually moved the file it the destination and gave a different error. This was quite weird - it worked! but gave some other error, and I can't replicate it now. You can see my confusion lol

[SilentVoid13]

Hey @frankreporting @mymykyta, sorry for the late response.

I wasn't able to reproduce this bug, could you try to update to Templater's latest version and see if the bug still exists in 1.9.9?

[frankreporting]

Now it's my turn to be sorry for the delayed response. I didn't get around to trying this out again until this weekend. Maybe it's been so long I've forgotten how I had all this set up, but I I can't seem to figure out where I had indicated the template I wanted to use on empty notes. Wasn't that a setting in Templater before? As of now, no templates at all are executed on new empty notes.

[frankreporting]

(UPDATE) Still not sure why I can't find the place to indicate a template for all empty notes, but anyway it looks like I can manually run my daily note template and it will successfully move the file without duplicating it, whether or not Obsidian Sync is enabled. Here's my updated script:

<%*
if (tp.file.folder(true) != "00_Meta/06_Logbook") { 
	await tp.file.move("/00_Meta/06_Logbook/" + tp.file.title) 
}
const displayDate = (datestr) => tp.date.now(datestr, 0, tp.file.title, "YYYY-MM-DD")
tR+="---\n"
tR+="alias: " + displayDate("M/D/YY dddd") + "\n"
tR+="---\n"
tR+="# " + displayDate("dddd, MMMM D, YYYY")
%>

[frankreporting]

OK, I can now report things are finally working as expected! Not sure if it was user error in the end or if the Templater update is what finally fixed it, but at some point, I did end up with a tp.file.move() command in both the empty note template AND the daily note template, and I'm fairly certain this is what was causing the duplicates after updating.

In any case, I was able to take advantage of the new approach using folder templates instead of an empty note template, and everything works beautifully. All new notes go to my inbox folder, and the inbox folder has a template that determines, based on the title, what kind of note it should be. In practice this means I can just use the Quick Switcher to see if the note exists and then, if not, create it using the appropriate template.

Ex. If I create a note @Person's Name Templater will now automatically apply my Person template, but if my new note is in the form 2021-10-31 it gets a daily note template, and 2021-10-31 Event title gets my Log template, etc.

Here's the code I'm using now in case anyone else finds it helpful..

Inbox folder template (which I call Get Template.md):

<%*
if (/^[0-9]{4}-[0-9]{2}-[0-9]{2}$/.test(tp.file.title)) {
	let dailyNote = await tp.file.include("[[Daily Note]]");
	tR += dailyNote;
} 
else if (tp.file.title.search(/^[0-9]{4}-[0-9]{2}-[0-9]{2} .*$/) !=-1) {
	let log = await tp.file.include("[[Log]]");
	tR += log;
} 
else if (/^@/.test(tp.file.title)) {
	let person = await tp.file.include("[[Person]]");
	tR += person;
} 
else if (/!task/.test(tp.file.title)) {
	let task = await tp.file.include("[[Task]]");
	tR += task;
} else
	{
		tR += "---\n" + "id: " + tp.date.now("YYYYMMDDHHmmssS") + "\n---";
	}
%>

And here's my Daily Note.md template, which you'll see also includes a Dataview object via customJS:

<%*
if (tp.file.folder(true) != "00_Meta/06_Logbook") { 
	await tp.file.move("/00_Meta/06_Logbook/" + tp.file.title) 
}
const displayDate = (datestr) => tp.date.now(datestr, 0, tp.file.title, "YYYY-MM-DD")
tR+="---\n"
tR+="alias: " + displayDate("M/D/YY dddd") + "\n"
tR+="---\n"
tR+="# " + displayDate("dddd, MMMM D, YYYY")
%>

<% tp.file.cursor(0) %>

# Log
\```dataviewjs
const {DvTasks} = customJS
DvTasks.dailyLog({app, dv, luxon, that:this})
\```

[frankreporting]

Oh no! It's back again, and inconsistent. Sometimes I create a note, it executes and moves the file to the appropriate folder. Other times, I get the old error that this file already exists in that location, and I see the new version has the rendered template, but there's a leftover empty note with the same filename sitting in my inbox.

Also, I'm seeing a new issue where if I access a note on mobile after I've already created it on desktop, it applies the template AGAIN. So I get template text appended twice at the top of the note. Not sure if that's been reported yet.

[shabegom]

Also, I'm seeing a new issue where if I access a note on mobile after I've already created it on desktop, it applies the template AGAIN. So I get template text appended twice at the top of the note. Not sure if that's been reported yet.

Likely a syncing issue unrelated to templater. If it is Templater please open a new issue.

Regarding the original issue, are you certain there isn't a file with the same name in the folder already?

[frankreporting]

It does sound like a possible sync issue. Should I report that on the Obsidian forum you think?

Regarding original issue, it's happened on very particular task or meeting note names, so the likelihood the note already existed is slim to none.

Will try to document some specific occurrences to share.

Likely a syncing issue unrelated to templater. If it is Templater please open a new issue.

Regarding the original issue, are you certain there isn't a file with the same name in the folder already?

[kalmir]

Hi, I am not sure if it is entirely related to this bug but I have had also the problem of Templater creating duplicate notes instead of moving them... and the solution was to put await before tp.file.move.

[welpdx]

Hi @kalmir, Thanks for your input but I think the error exists with await in front tp.file.move. In fact, the original post has it as await tp.file.move already. I think the problem is something else entirely. But I do like your reminder to put await there because sometimes I do forget lol.

[frankreporting]

OK, I don't want to speak too soon, but it looks like the latest insider update Obsidian 0.14.5 may have fixed the duplicate issue by improving Obsidian Sync. I'll keep an eye on it but I'm having success on both mobile and desktop.

[welpdx]

oooooh that is good to hear