[turborepo] failed to contact turbod/timed out waiting for initial fsevents cookie: filewatching failed to start

[ambroselittle]

What version of Turborepo are you using?

1.9.1

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Describe the Bug

Per @nathanhammond's request here, opening a new issue. See environmental details in "To Reproduce."

When running a command, I get:

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
        - the process identified by the pid in the file at /var/folders/7t/8fc_x2f96k3fq2w07f_y19xh0000gp/T/turbod/9ad75a386d6cc29e/turbod.pid is not running, and remove /var/folders/7t/8fc_x2f96k3fq2w07f_y19xh0000gp/T/turbod/9ad75a386d6cc29e/turbod.pid
        - check the logs at /Users/myuser/Library/Application Support/turborepo/logs/9ad75a386d6cc29e-mfe-utilities.log
        - the unix domain socket at /var/folders/7t/8fc_x2f96k3fq2w07f_y19xh0000gp/T/turbod/9ad75a386d6cc29e/turbod.sock has been removed
        You can also run without the daemon process by passing --no-daemon

Daemon process/socket folder doesn't exist (right after running the turbo command). But if I run turbo daemon start it creates a folder there, but it is empty.

Status:

> turbo daemon status
ERROR daemon is not running

Then:

> turbo daemon start
ERROR timeout while watchin directory: deadline has elapsed

> turbo daemon 
2023-04-17T12:29:24.780-0400 [ERROR] turbod: error watching /Volumes/Code/Repos/Org/my-repo: timed out waiting for initial fsevents cookie: filewatching failed to start
Turbo error: watching /Volumes/Code/Repos/Org/my-repo: timed out waiting for initial fsevents cookie: filewatching failed to start

Note: This is on a secondary APFS (Case-sensitive) volume called Code. (I do this to avoid issues I've seen in past where code only breaks on server that is case sensitive.) Someone on the other issue mentioned this as well (and others after mentioned similar).

As suggested, tried npx create-turbo@latest with same results.

Log:

2023-04-17T12:34:33.127-0400 [ERROR] turbod: error watching /Volumes/Code/Repos/Org/my-turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start
2023-04-17T12:34:34.124-0400 [ERROR] turbod: error watching /Volumes/Code/Repos/Org/my-turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start
2023-04-17T12:34:35.122-0400 [ERROR] turbod: error watching /Volumes/Code/Repos/Org/my-turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start
2023-04-17T12:34:51.350-0400 [ERROR] turbod: error watching /Volumes/Code/Repos/Org/my-turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start

If I run the same on my primary partition, all is hunky dory. It creates the folder and the .pid and .sock are there.

Unsurprisingly (perhaps), both locations resolve to same:

> which turbo
/Users/ambrose.little/Library/pnpm/turbo

> turbo --version
1.9.1

Tried pnpm remove -g turbo and re-install. Same result. Running sudo turbo daemon start also fails the same way.

Expected Behavior

Should work/run the daemon without error. :)

To Reproduce

OS + Version: 13.3.1 (22E261) turbo version: 1.9.1 (also saw on 1.8.3/8) pnpm: 8.1.1 (also with 7.31.0) Shell: see same results in VS Code and Warp, using oh my zsh Repository contents: can repro with npx create-turbo@latest Repo filesystem permissions (on both the primary partition that works and secondary that doesn't):

drwxrwxr-x ambrose.little staff

The only distinction I can see is that it's on the external partition.

Reproduction Repo

No response

[ph55]

Will duplicate my comment from related issue (might be helpful) https://github.com/vercel/turbo/issues/2034#issuecomment-1501195480

MacOS Ventura 13.2.1 Looks like permissions issue. I can start the daemon with sudo. Without the sudo - no success.

Another weird thing. Running turbo from nested project directory (no sudo) throws ERROR timeout while watchin directory: deadline has elapsed error:

But running it from "home" directory gives no error:

Demon pid is diffrrent from the first example (tmp folder)

[mehulkar]

We've made some improvements here, are you still seeing this on the latest version?

[QuiiBz]

We're having the same issue with our monorepo at work when upgrading from 1.10.14 to 1.10.16:

 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed.
        To quickly resolve the issue, try running:
        - $ turbo daemon clean

        To debug further - please ensure the following:
        - the process identified by the pid in the file at /tmp/turbod/46ebc33c7aff93d3/turbod.pid is not running, and remove /tmp/turbod/46ebc33c7aff93d3/turbod.pid
        - check the logs at /root/.local/share/turborepo/logs/46ebc33c7aff93d3-shire.log
        - the unix domain socket at /tmp/turbod/46ebc33c7aff93d3/turbod.sock has been removed

        You can also run without the daemon process by passing --no-daemon

As expected, --no-daemon works fine. I've tried debugging a bit more by running the following commands and checking the pid, UNIX socket, and logs files (I'm running as root):

❯ pnpm turbo daemon status
Turbo error: unable to connect: daemon is not running
❯ pnpm turbo daemon start
Turbo error: unable to connect: timeout while watchin directory: deadline has elapsed
❯ pnpm turbo daemon clean
Done
❯ pnpm turbo daemon start
Turbo error: unable to connect: timeout while watchin directory: deadline has elapsed
❯ cat /tmp/turbod/46ebc33c7aff93d3/turbod.pid
'/tmp/turbod/46ebc33c7aff93d3/turbod.pid': No such file or directory (os error 2)
❯ cat /tmp/turbod/46ebc33c7aff93d3/turbod.sock
'/tmp/turbod/46ebc33c7aff93d3/turbod.sock': No such file or directory (os error 2)
❯ cat /root/.local/share/turborepo/logs/46ebc33c7aff93d3-shire.log
2023-10-17T19:26:14.809+0200 [ERROR] turbod: error turbod shut down from inactivity
2023-10-17T19:26:14.811+0200 [INFO]  turbod.rpc server.FileWatcher: Events channel closed. Exiting watch loop
2023-10-17T19:26:14.811+0200 [WARN]  turbod.rpc server.GlobWatcher: GlobWatching is closing due to file watching closing
2023-10-20T21:22:50.124+0200 [ERROR] turbod: error turbod shut down from inactivity
2023-10-20T21:22:50.126+0200 [INFO]  turbod.rpc server.FileWatcher: Events channel closed. Exiting watch loop
2023-10-20T21:22:50.126+0200 [WARN]  turbod.rpc server.GlobWatcher: GlobWatching is closing due to file watching closing
2023-10-23T10:06:37.757+0200 [INFO]  turbod.rpc server.FileWatcher: Events channel closed. Exiting watch loop
2023-10-23T10:06:37.757+0200 [WARN]  turbod.rpc server.GlobWatcher: GlobWatching is closing due to file watching closing

[ambroselittle]

@mehulkar It seems to have fixed the reported issue. I now see this when building for each package: Failed to mark outputs as cached for @mypkg/my-utility#build: rpc error: code = DeadlineExceeded desc = failed to load filewatching in time

[ambroselittle]

I'm on 1.11.2 now. Tried again.

turbo daemon      
 ERROR  filewatching failed to start: filewatching failed to start: waiting for cookie timed out: deadline has elapsed

turbo daemon status
Turbo error: unable to connect: daemon is not running

turbo daemon start 
Daemon is running

Immediately after that:

turbo daemon restart
Turbo error: unable to connect: daemon is not running

Building is faster as noted before (even w/o the no daemon flag), but get warnings as below.

pnpm build

See a few of these (per package): WARNING Failed to check if we can skip restoring outputs for @built/core-utility#build: GrpcFailure(DeadlineExceeded). Proceeding to check cache Failed to mark outputs as cached for @built/core-utility#build: GrpcFailure(DeadlineExceeded)

@mehulkar, let me know if you want me to try anything.

[jindong-zhannng]

Any update? I am facing same problem on WSL2 Ubuntu environment.

[tyleralbee]

I originally went to post this in #7413 but it sounds like @arlyon determined the root cause to be related to this issue.

When trying to install the Turborepo LSP VSCode extension on Ubuntu 22.04, I'm running into what I believe to be more than one issue.

Note that I recently upgraded to 2.0.3 following the guide using the codemod.

Upon installing, I get three errors:

1. First:

{"code":127,"killed":false,"signal":null,"cmd":"/home/redacted/.local/share/pnpm/turbo\n daemon start"}

2. Second:

Internal error

3. Third (last):

Server initialization failed

I have turbo installed globally:

which turbo
# /home/redacted/.local/share/pnpm/turbo

It's not able to start the daemon because it can't find the turbo binary, which works just fine for me:

turbo daemon start
# ✓ daemon is running

It's looking in the right place, which makes me wonder if the cmd path is not resolving how it's expected to? E.g.:

/home/redacted/.local/share/pnpm/turbo\n daemon start
# bash: /home/redacted/.local/share/pnpm/turbon: No such file or directory

I am unsure if this is a separate problem from the server initialization failing, too:

[Info  - 2:57:08 AM] root uri: /tmp/turbod/bfea02dea249cc76/turbod.sock
[Error - 2:57:08 AM] failed to connect to daemon: unable to make handshake: version mismatch: Some("version mismatch. Client 1.11.0 Server 2.0.0")
[Error - 2:57:08 AM] Server initialization failed.
  Message: Internal error
  Code: -32603 
[Error - 2:57:08 AM] Turborepo Language Server client: couldn't create connection to server.
  Message: Internal error
  Code: -32603 

[arlyon]

Hey @tyleralbee I can fix at least one of those. Will push a release of the LSP in the next hour or so, and ask you to try again. Thanks for being so detailed!

[nike1v]

I'm still getting all of this erors when using Turborepo LSP ext for vscode.

[Info  - 5:34:27 PM] root uri: /tmp/turbod/1eb4113d5003ac15/turbod.sock
[Error - 5:34:34 PM] failed to connect to daemon: timeout while watching directory: deadline has elapsed
[Error - 5:34:34 PM] Server initialization failed.
  Message: Internal error
  Code: -32603 
[Error - 5:34:34 PM] Turborepo Language Server client: couldn't create connection to server.
  Message: Internal error
  Code: -32603 

I use it under WSL2.

[kmalloy24]

Here from https://github.com/vercel/turbo/issues/7413.

v2.0.0 of the VS Code LSP extension isn't working for me on Windows 11.

Whenever I enable it I get these errors.

[YBouz]

Here from #7413.

v2.0.0 of the VS Code LSP extension isn't working for me on Windows 11.

Whenever I enable it I get these errors.

Same problem here