Deletion of pre-existing route folders in `urara.clean()`

[nberlette]

https://github.com/importantimport/urara/blob/b5520f925fea031fc04d672dbb2f8a40fcddf48d/urara.js#L86-L96

I noticed that pre-existing folders in the routes directory are blown away by the urara.clean function, as you can see in the section of code highlighted above. Urara apparently just deletes all routes folders arbitrarily, without checking if it was a route that it created or not.

This means if I had a subfolder for photos, for example, after running the dev server urara would destroy them and not even realize it made a mistake. This has huge implications down the line. Unfortunately I don't have the bandwidth to draft a PR right now, but I'll still try to squeeze it in.

That being said: wonderful project you have here so far, I'm quite eager to try it in production as soon as possible 😉

[kwaa]

🤔 hmm... my suggestion is to put all the folders in the /urara/ directory, but this really needs improvement. I may update in the next few days, or welcome PR 👌

[nberlette]

I'll try to put something together this weekend 😅

[kwaa]

progress: It is currently possible to delete all files copied from /urara/, but it is difficult to delete all empty directories.

const cleanFile = src =>
    fs.readdir(src, { withFileTypes: true }).then(files => {
      files.forEach(file => {
        const dest = path.join(src, file.name)
        file.isDirectory()
          ? cleanFile(dest)
          : file.name.startsWith('.')
            ? log('cyan', 'ignore file', dest)
            : rmFile(dest)
      })
  })
cleanFile('urara')

[nberlette]

What if it kept track of all the files/dirs copied in the build step? Maybe in some sort of context or an array that can keep state? That would remove all the guesswork and be quite precise, all it would need to do is iterate through the array and delete items that match.

[kwaa]

What if it kept track of all the files/dirs copied in the build step? Maybe in some sort of context or an array that can keep state? That would remove all the guesswork and be quite precise, all it would need to do is iterate through the array and delete items that match.

💀 It would be too complicated in this case. the temporary solution is to delete all files and folders copied from /urara/, disadvantage is that cannot use /src/routes/xxx/yyy.svelte and /urara/xxx/zzz.svelte.md at the same time (if both are placed in the urara directory, then they can)

const cleanDir = src =>
  fs.readdir(src, { withFileTypes: true }).then(files => {
    files.forEach(file => {
      const dest = path.join(src, file.name)
      file.isDirectory()
        ? rmDir(dest)
        : file.name.startsWith('.')
          ? log('cyan', 'ignore file', dest)
          : rmFile(dest)
    })
  })

[kwaa]

Update: I tried to implement the scanDir() function to create a list of files and so far it works fine for copying but removing is still hard to do. files are in kwaa/blog/urara.js

[NullFragment]

What's the problem that you're running into with file removal? I might have a chance to look at it this week.

Just after looking briefly at the code I had a few ideas jump out:

• One option would be to scan src/routes before a build, and create a routes.lock (or something) file with the current file structure. Then you could just load that during a clean and not delete anything in it. The main disadvantage of that route is you'd still delete anything that was created since the last clean
• Is it possible to change the directory of the files that are copied from src/routes to src/routes/generated or something like that? In that case you could just avoid the whole scanning mess altogether.
• Add a prefix/suffix to all copied files/dirs (maybe the current epoch time?) and write that out to a lockfile. Then you'd have an easy regex filter on your clean method.

edited: after re-reading, the first and second options I originally suggested were basically the same thing, just the first was more complicated.

[kwaa]

What's the problem that you're running into with file removal? I might have a chance to look at it this week.

Just after looking briefly at the code I had a few ideas jump out:

• One option would be to scan src/routes before a build, and create a routes.lock (or something) file with the current file structure. Then you could just load that during a clean and not delete anything in it. The main disadvantage of that route is you'd still delete anything that was created since the last clean
• Is it possible to change the directory of the files that are copied from src/routes to src/routes/generated or something like that? In that case you could just avoid the whole scanning mess altogether.
• Add a prefix/suffix to all copied files/dirs (maybe the current epoch time?) and write that out to a lockfile. Then you'd have an easy regex filter on your clean method.

edited: after re-reading, the first and second options I originally suggested were basically the same thing, just the first was more complicated.

Hello, my idea is to keep it simple while solving the problem, so the newest solution is to scan the files under the urara folder to generate a temporary index while copying/cleaning, and copy or remove them in depth order.

Copying goes well, the problem is with cleanup - when it starts to determine if the folder is empty, the files that need to be removed have not yet been removed.

[NullFragment]

Ah, interesting. Things have been crazy with work, but I think I'll have some time Thursday to look at it. My thought was to see if it'd be viable to create two arrays (one for dirs and one for files within the scan method. That way it should be possible to iterate through all of the files schedule those to be deleted async first, wait for them to all be deleted, then remove the directories.

[NullFragment]

So, I was playing around with the javascript file a bit, but decided to just start over from scratch using typescript, at least for the file management part just because it's a bit more familiar. It's not the prettiest solution but it works and I generally just prefer the clarity of functions over assigning lambdas. I wrote the method from scratch, then I found node-dir and added a little wrapper around it to give depth info, so there are options.

Anyways, here's a gist with the output (it's similar for both). There's some logging to clean up or move over to .debug in there. https://gist.github.com/NullFragment/ccb1cfb5fda14d904f82e2f0ce09136d

The output of either method is similar, except that the node-dir one uses os-specific path separators (/ vs ).

Filling out the rest of the copy and removal should be pretty easy, I just wanted to make sure you'd be comfortable with the solution before I finish implementing it.

[kwaa]

So, I was playing around with the javascript file a bit, but decided to just start over from scratch using typescript, at least for the file management part just because it's a bit more familiar. It's not the prettiest solution but it works and I generally just prefer the clarity of functions over assigning lambdas. I wrote the method from scratch, then I found node-dir and added a little wrapper around it to give depth info, so there are options.

Anyways, here's a gist with the output (it's similar for both). There's some logging to clean up or move over to .debug in there. https://gist.github.com/NullFragment/ccb1cfb5fda14d904f82e2f0ce09136d

The output of either method is similar, except that the node-dir one uses os-specific path separators (/ vs ).

Filling out the rest of the copy and removal should be pretty easy, I just wanted to make sure you'd be comfortable with the solution before I finish implementing it.

Great work! just don't really want to add node-dir to the dependencies, I'll try it in a few hours.

[NullFragment]

Yeah, I figured I’d leave both options there. One thing I thought of is adding a dry-run option for clean that would print out what it would delete before you go ahead and run it.

[kwaa]

It works fine, I will try to combine it with the existing remove function tomorrow.

Yeah, I figured I’d leave both options there. One thing I thought of is adding a dry-run option for clean that would print out what it would delete before you go ahead and run it.

Considering that urara.js will only handle files copied from the /urara/ folder, this doesn't seem necessary...

[kwaa]

😥 Ok, now the problem comes back to removing dirs... The current remove function is still problematic, and I can't think of a better way to do this.

[NullFragment]

Sorry for the delayed response. I missed the message about removal. I have a few questions:

• Are you trying to remove the dirs in parallel? If you are, you need to make sure you get a sorted list of the keys and go in reverse order since the dirs and files structures are organized by depth. So you'd have a serial loop that spawns parallel deletes at each depth, starting from the deepest. The files shouldn't matter because they're not dependent on the hierarchy, but the dirs are.
• Is it even worth parallelizing this? I'm not sure that there would be much benefit to parallelization until you hit a large number of files.
• Is copying even necessary or could symlinks just be used for the directories?

[kwaa]

Sorry for the delayed response. I missed the message about removal. I have a few questions:

• Are you trying to remove the dirs in parallel? If you are, you need to make sure you get a sorted list of the keys and go in reverse order since the dirs and files structures are organized by depth. So you'd have a serial loop that spawns parallel deletes at each depth, starting from the deepest. The files shouldn't matter because they're not dependent on the hierarchy, but the dirs are.
• Is it even worth parallelizing this? I'm not sure that there would be much benefit to parallelization until you hit a large number of files.
• Is copying even necessary or could symlinks just be used for the directories?

Ah, no problem.

• I reverse the Map and use forEach to remove. Since the remove function is not modified, the same problem does occur... I'm not very good with files.

• hmm... It's just that I wanted to use fs/promises, but some blogs can indeed have very much files.

• I haven't tried it. Maybe it's worth a try?