windows: Publishing with quartz sync and symlinks fails

[reallyely]

Describe the bug

With a quartz app set up with it's content hosted as a symlink, publishing to Github Pages results in an empty app.

To Reproduce Steps to reproduce the behavior:

1. Create a new Empty Quartz app
2. Create content folder with symlink New-Item -Path content -ItemType SymbolicLink -Value C:\path\to\vault\<vault>
3. run npx quartz sync

C:\dev\quartz [main ↑1 +1 ~0 -0 !]> New-Item -Path content -ItemType SymbolicLink -Value C:\vault 
C:\dev\quartz [main ↑1 +1 ~0 -0 !]> npx quartz sync --no-pull

 Quartz v4.2.3

Backing up your content
Detected symlink, trying to dereference before committing
warning: adding embedded git repository: content
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint:
hint:   git submodule add <url> content
hint:
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint:
hint:   git rm --cached content
hint:
hint: See "git help submodule" for more information.
hint: Disable this message with "git config advice.addEmbeddedRepo false"
[main a6ba7fc] Quartz sync: Jul 12, 2024, 10:05 AM
 1 file changed, 1 insertion(+)
 create mode 160000 content
Pushing your changes
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 411 bytes | 411.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/really-ely/notes.git
   bc28eb5..a6ba7fc  main -> main
branch 'main' set up to track 'origin/main'.
Done!

Git remote updates Actions run successfully Pages URL is published

Expected behavior

• Content Folder on remote to have content
• GitHub Pages URL generated from Actions to navigate to index

Actual

• Content folder resembles submodule in remote
• visiting published URL displays index.xml. Other pages return 404
• Local symlink folder displays as empty but admin user no longer has permissions to read it

Desktop (please complete the following information):

• Quartz Version: e.g. v4.2.3
• node Version: v18.19.0
• npm version: v10.7.0
• OS: Windows 11
• Browser Brave

Additional context

• Powershell 5.1
• deploy.yml copied from documentation
• branch changed from v4 to main
• The Obsidian vault that I was trying to symlink to from Quartz was a OneDrive folder

[reallyely]

• I manually copied my vault over to the content folder to see if I could get it to deploy in a more direct use case. It was successful.
• I then locally deleted the quartz/content folder
• I moved my Obsidian vault out of my one drive folder, created a new sym link and re-ran npx quartz sync.
• The remote still had a static content folder
  • it's contents showed last updated at the time of my last push
  • Updated contents from the symlinked folder now also showed up
  • The website rendered as expected with updated content

However, the content folder in my quartz directory is still unreachable after the sync

C:\dev\quartz [main ≡ +1 ~0 -61 !]> ls content


    Directory: C:\dev\quartz


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---l         7/12/2024  11:02 AM              0 content

C:\dev\quartz [main ≡ +1 ~0 -61 !]> cd content

cd : Cannot find path 'content' because it does not exist.
At line:1 char:1
+ cd content
+ ~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (content:String) [Set-Location], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand

[reallyely]

I thought I'd share my solution to this problem that directly serves my needs. It's opinionated but may help someone else.

I chose to skip using quartz sync and write my own solution. I'm still using the deploy.yml that is in the getting started guide, but I created a specific branch for publishing called publish. The general strategy is to locally check out a branch to copy the symlinked folder contents in, pushing that and letting the build run. I then switch back to my original branch and delete the local publish. I force push to publish with the assumption that I always want the latest state and don't want to resolve conflicts.

Additionally, I wanted it to continuously watch for changes and publish at regular intervals to keep things immediately up to date. I'm a noob at this file management stuff so my hashing attempt may be naive and costly but it's working for my small garden.

I like that this keeps main focused on the application, and publish on the contents and we don't need to worry about restoring the symlinked folder.

For context, I have had an Obsidian vault for many years now and wanted a permanent solution for sharing those core notes with separate vaults I create for individual jobs. Usually those job-related ones need to be locked down in some way and live and die on their infrastructure so I symlink my core notes into the job.

With this script I'm able to work seamlessly between both vaults and publish all of that to the web continuously.

[reallyely]

import assert from "assert"
import crypto from "crypto"
import { execSync } from "child_process"
import fs from "fs-extra"
import { hideBin } from "yargs/helpers"
import ora from "ora"
import path from "path"
import process from "process"
import yargs from "yargs"

interface Arguments {
  interval: number
}
const argv = yargs(hideBin(process.argv))
  .option("interval", {
    alias: "i",
    description: "Interval in minutes between checks",
    type: "number",
    default: 5,
  })
  .help()
  .parse() as Arguments

const checkIntervalMinutes = argv.interval as number
const checkIntervalMs = checkIntervalMinutes * 60 * 1000
const contentPath: string = "./content"
const publishBranch: string = "publish"
// State used to detect when we need to update
let previousHash = ""
let lastCheckTime = new Date()
let lastPublishTime: Date | null = null
let publishStatus = "Not published yet"

const spinner = ora({
  text: "Initializing...",
  color: "yellow",
  spinner: "dwarfFortress",
}).start()

function updateSpinner(text: string, color: "yellow" | "red" = "yellow") {
  spinner.color = color
  spinner.text = text
}
function persistMessage(message: string, error = false) {
  const timestamp = new Date().toLocaleTimeString()
  spinner.stopAndPersist({
    symbol: error ? "❌" : "✔",
    text: `[${timestamp}] ${message}`,
  })
  spinner.start()
}

function execCommand(command: string): string {
  return execSync(command, { encoding: "utf8", stdio: "pipe" }).trim()
}

async function main(): Promise<void> {
  const originalBranch = currentBranch()

  // Step 4: Switch to publish branch
  updateSpinner("Switching to publish branch")
  execCommand(`git checkout -B ${publishBranch}`)

  // Step 1: Check if content is a symlink
  updateSpinner("Checking content folder")
  const contentStats = await fs.lstat(contentPath)
  if (!contentStats.isSymbolicLink()) {
    throw new Error("content is not a symbolic link")
  }

  // Step 2: Get the target of the symlink
  const linkTarget = await fs.readlink(contentPath)

  // Step 3: Create a perfect copy
  updateSpinner("Creating copy of content")
  const tempPath = `${contentPath}_temp`
  await fs.copy(linkTarget, tempPath, {
    dereference: true,
    preserveTimestamps: true,
  })

  // Step 5: Replace symlink with actual folder
  updateSpinner("Replacing symlink with actual folder")
  await fs.remove(contentPath)
  await fs.move(tempPath, contentPath)

  // Step 6: Commit changes
  updateSpinner("Committing changes")
  const date = new Date().toISOString().split("T")[0]
  const time = new Date().toTimeString().split(" ")[0]
  execCommand(`git add ${contentPath}`)
  execCommand(`git commit -m "Content update: ${date} ${time}"`)

  // Step 7: Push changes to publish branch
  updateSpinner("Pushing changes to publish branch")
  execCommand(`git push -f origin ${publishBranch}`)

  // Step 8: Switch back to original branch
  updateSpinner("Switching back to original branch")
  execCommand(`git checkout ${originalBranch}`)

  assert.strictEqual(
    currentBranch(),
    originalBranch,
    `Failed to return to original branch. Expected ${originalBranch}, but on ${currentBranch}`,
  )

  updateSpinner(`Deleting ${publishBranch} branch`)
  execCommand(`git branch -D ${publishBranch}`)

  // // Step 9: Restore symlink
  // spinner.text = "Restoring symlink"
  // await fs.remove(contentPath)
  // await fs.symlink(linkTarget, contentPath)

  // Step 10: Assert content is a symlink again
  const finalContentStats = await fs.lstat(contentPath)
  if (!finalContentStats.isSymbolicLink()) {
    throw new Error("content is not a symbolic link after restoration")
  }

  // Step 11: Check if symlink is accessible
  await fs.access(contentPath)
}

function generateContentHash(dir: string): string {
  const files = fs.readdirSync(dir)
  let hash = crypto.createHash("md5")

  for (const file of files) {
    const filePath = path.join(dir, file)
    const stats = fs.statSync(filePath)

    if (stats.isDirectory()) {
      hash.update(generateContentHash(filePath))
    } else {
      hash.update(`${file}:${stats.mtime.getTime()}`)
    }
  }

  return hash.digest("hex")
}

async function checkAndSync() {
  updateSpinner("Checking for changes...")

  const currentHash = generateContentHash(contentPath)
  lastCheckTime = new Date()

  if (currentHash !== previousHash) {
    updateSpinner("Changes detected, syncing content...")
    try {
      await main()
      lastPublishTime = new Date()
      publishStatus = "Success"
      persistMessage("Content synced successfully")
    } catch (error) {
      publishStatus = "Failed"
      persistMessage(`Content sync failed ${error}`, true)
    }
    previousHash = currentHash
  }

  updateDisplay()
}

function updateDisplay() {
  const checkTime = lastCheckTime.toLocaleTimeString()
  const publishTime = lastPublishTime ? lastPublishTime.toLocaleTimeString() : "N/A"

  spinner.text = `
⌜------------------⌝
| Last Check    🕵️ : ${previousHash ? "Changes Detected" : "No Changes"}  ${checkTime}
|------------------:
| Last Publish  🚀 : ${publishStatus}  ${publishTime}
⌞------------------⌟
`
}

function currentBranch() {
  return execCommand("git rev-parse --abbrev-ref HEAD")
}

const checkInterval = setInterval(checkAndSync, checkIntervalMs)

process.on("SIGINT", () => {
  clearInterval(checkInterval)
  spinner.stop()
  console.log("\nScript terminated by user")
  process.exit(0)
})

checkAndSync()

[quakeboy]

Same happens to me too, I assumed git would handle it as a folder and copy the contents. Seems like I am wrong. What could be an easy solution for an ex-developer (but not web)?

I want something where I can update my github repo from either my desktop or macbook. I already put the contents on icloud drive, so I have the obsidian vaults in both computers.

Is there a different way?

[chestNutLsj]

TLDR: use hard link!

This problem is happening on my side as well. In the past I was using Arch Linux as my development environment, and when I used symlink directly to obsidan vault there, I was able to commit correctly to the remote repository on github via git (the root cause of the npx quartz sync failure). However, when I switched to windows11 as the development environment, the git commit to symlink only has the path to the target indexed folder, not the folder and its contents. After searching and thinking about it, I'm sure it's because the implementation of symlink is different under different platforms, so git will also have the same strange problem, if you are a Chinese developer, you can check out this link to learn more about the cause: https://geek-docs.com/git/git-questions/69_git_how_do_i_commit_a_windows_symlink_into_git.html .

Now I'll tell you my solution (valid 2024.09.10):

1. when creating a quartz app, first create an empty app,
2. then check if there is a content folder in this directory, if there is, then delete it, otherwise use the mklink -J content /path/to/obsidian/valut command to create a hard link to the obsidan library.

After that you can properly commit using git.