turbo
turbo copied to clipboard
vercel
WARNING failed to contact turbod.
What version of Turborepo are you using?
1.5.1
What package manager are you using / does the bug impact?
npm
What operating system are you using?
Windows
Describe the Bug
After upgrading from 1.4.7 I now get a warning. Following the warning text leads to not a fix. The log is fine and I've removed the temp files, no fix.
Expected Behavior
No warning or turbod properly launching. If a warning must be shown, it should reflect failure to launch turbod, not contact it.
To Reproduce
Upgrade a 1.4.7 to 1.5.1
In 1.5 we have toggled on using the daemon by default. On your machine the connection between the main turbo process and the daemon is failing.
This does not prevent your build from running to completion but we do need to identify why that connection is failing.
- Can you tell me more about your OS?
- Can you identify if you have write permissions to the path generated here? https://github.com/vercel/turborepo/blob/0dfa8e681c64a2bba9e8227d349b87f2f4bf7d04/cli/internal/daemon/daemon.go#L47 (we need better logging here to make this easier, you'd need to identify where Golang tempdir is creating a folder)
- what permissions does your user have?
@nathanhammond Thanks for the reply. How do you even start the daemon? I searched the docs and only the cache config mentions a daemon. I understand the principle, but I just upgraded, didn't make a change to turbo.json to run a daemon. Is this something like Gradle where it spins it up on first launch? Just wanted to mention that some confusion for me is the lack of docs.
I also found the option --no-daemon to turn the warning off. The issue though is that the process identified in turbo.pid doesn't exist. Not running. So, maybe it tries to launch a daemon and can't?
* Can you tell me more about your OS?
Edition Windows 10 Pro for Workstations Version 21H2 Installed on 11/24/2021 OS build 19044.2006 Experience Windows Feature Experience Pack 120.2212.4180.0
* Can you identify if you have write permissions to the path generated here? https://github.com/vercel/turborepo/blob/0dfa8e681c64a2bba9e8227d349b87f2f4bf7d04/cli/internal/daemon/daemon.go#L47 (we need better logging here to make this easier, you'd need to identify where Golang tempdir is creating a folder)
%HOME%\AppData\Local\Temp full permission. Also turbo creates a turbod folder there and turbod.pid and turbod.sock.
* what permissions does your user have?
For me that hexHash join is Temp\turbod\5ad724522893520a
I am fairly certain I have full permission. I run other go executables and go tools without issue. I can also open and edit the pid file. The sock file is 0 bytes and the pid file has a pid, but when I run that pid is not running, exluding --no-daemon. This makes the warning inaccurate. The log shows no error.
It should be WARNING: Failed to start turbod, because that seems to be what is happening. Something needs to go into the log to say why, which is also not happening.
turbo daemon status should give a location for logs from the daemon. If you could post those, it would be helpful to figure out why the daemon is not running for is unable to be contacted.
Hi there, also getting the same issue. On trying turbo daemon status I get this:
❯ pnpm exec turbo daemon status
Failed to contact daemon: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /tmp/turbod/0f771c1e12c2b3b3/turbod.pid is not running, and remove /tmp/turbod/0f771c1e12c2b3b3/turbod.pid
- check the logs at /home/jayg-hive/.local/share/turborepo/logs/0f771c1e12c2b3b3-Prosper-Ultimate.log
- the unix domain socket at /tmp/turbod/0f771c1e12c2b3b3/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
I'm on WSL2 with the following details:
WSL version: 0.67.6.0
Kernel version: 5.15.62.1
WSLg version: 1.0.44
MSRDC version: 1.2.3401
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.601
Nothing out of the ordinary on the logs, they just say something about fsnotify events on my monorepo files:
2022-09-23T09:27:16.339+0800 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {/home/jayg-hive/hive-gaming/**redacted**/packages/utils/.turbo/turbo-test.log 3}
Same issue for me on Mac, today installed turbo for first time
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid is not running, and remove /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.pid
- check the logs at /Users/marius/Library/Application Support/turborepo/logs/5c8a39790d3101dc-test__turborepo.log
- the unix domain socket at /var/folders/b2/7xflcg550111jtksj2sb32n00000gn/T/turbod/5c8a39790d3101dc/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
Error from log is
2022-09-23T10:48:48.558+0700 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Media/Dev Projects/Projects/test__turborepo: timed out waiting for initial fsevents cookie: filewatching failed to start"
Gets the same warning message as @MariuzMM and @jayg-hive
Been using turbo for a while now. This warning started in version 1.5.2
On Windows10:
2022-09-23T14:16:10.471+0200 [ERROR] turbod: error: EXTRA_VALUE_AT_END="failed to lock the pid file at . Is another turbo daemon running?: Locked by other process"
No error on WSL2/Ubuntu :)
I'm seeing the same after a sweeping upgrade of dependencies today. Log output is a repeat of:
2022-09-23T09:42:53.645-0400 [ERROR] turbod: error: EXTRA_VALUE_AT_END="watching /Volumes/Work/code/MYPROJECT: timed out waiting for initial fsevents cookie: filewatching failed to start"
Console is outputting:
yarn turbo daemon status
yarn run v1.22.19
$ /Volumes/Work/code/MYPROJECT/node_modules/.bin/turbo daemon status
Failed to contact daemon: the daemon is not running
✨ Done in 0.32s.
Reverting and hard-coding to "turbo": "1.5.1" in package.json "solves" this issue.
Couple things going on:
- Minor, display-only bug related to
EXTRA_VALUE_AT_END. I'll fix it, but that part is safe to ignore - re: cookie timeouts, I'm going to bump it a little, but if it takes too long, it's likely better to for us skip filewatching.
- on that topic, the warning should be downgraded to informational. Filewatching is an optional optimization, and is not necessary for the proper functioning of
turbo.
For those of you hitting the warning, can you confirm that turbo is still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.
@gkn if you run turbo daemon status, it should give you the path to the .pid file? Can you check if there is in fact a process with that PID? Anything in the daemon logs (also from turbo daemon status) ?
Running MacOS 12.6 here. When running the command turbo daemon status, I get "Failed to contact daemon: the daemon is not running." It's not preventing building for me, just giving the warning.
Additional context and debugging thoughts:
-
turbo daemon statusis not intended to start the daemon. If it's not running, that's the status and that's fine. The daemon also does not stay alive forever, it has an idle timeout. Theturbocli will attempt to restart it next time it needs it. - running
turbo daemonin your monorepo root will run the daemon in the foreground and log to the terminal. - There are some additional daemon-management commands: restart, start, and stop. See
turbo daemon --help
Also, to everyone, thanks for your patience while we iron out the edge cases on this. There may be platforms and scenarios where the daemon does not work or does not provide benefit, and we'll work to detect those and behave accordingly as we identify them.
Hopefully this will help. I run the daemon and no errors to the console etc. Looks good using turbo daemon.
When I run turbo run dev the turbod dies. Last log message is
2022-09-23T14:04:32.896-0700 [DEBUG] turbod.rpc server.GlobWatcher: Got fsnotify event {oranges\apps\web\.turbo\turbo-lint.log451477266 4}
Couple things going on:
- Minor, display-only bug related to
EXTRA_VALUE_AT_END. I'll fix it, but that part is safe to ignore- re: cookie timeouts, I'm going to bump it a little, but if it takes too long, it's likely better to for us skip filewatching.
- on that topic, the warning should be downgraded to informational. Filewatching is an optional optimization, and is not necessary for the proper functioning of
turbo.For those of you hitting the warning, can you confirm that
turbois still producing the correct results, just with the extra warning? Or is this actively breaking anyone? If so, I'd like to prioritize addressing breakage.
Yup, no breakage whatsoever. It still runs the intended task. 👍
On Windows 10:
- I cleared the logs and pid + cookie file and folders in the user folder.
I still get the same warning message but now get a log file without error messages, just a lot of fsnotify messages. It even includes folders that is not specified as part of npm workspaces. This seems odd and wasteful, especially since it will watch node_modules in that folder too:
2022-09-24T08:41:40.051+0200 [DEBUG] turbod.rpc server.fsnotify: watching directory C:\xxxx\tools\storybook\node_modules@mrmlnc
Initial startup seems slow.
Monorepo is relatively large, a few hundred folders with roughly 5000 source files.
-
Tried manual start of turbod: $ npx turbo daemon start connection to turbo daemon process failed. Please ensure the following: ... ...
-
Size of log file
I primarily use WSL2/Ubuntu for this project. I noticed that the size of the turbod log file was large. (roughly 1.5 million lines, 260 MB) in just a few turbo runs since yesterday. Seems odd.
Turning down daemon logging: #2085
@gkn Thanks for the details. Better scoping filewatching sounds like an area for improvement.
Everything still seems to work fine, but it throws warnings on every run.
Same here, I'm running [email protected] on arm macos 12.5.1.
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid is not running, and remove /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.pid
- check the logs at /Users/Andrzej_Klapec/Library/Application Support/turborepo/logs/fe25f7293c19b4d8-tui-frontend-monorepo.log
- the unix domain socket at /var/folders/ls/sq7g081s1g3cvtlz41qp8z2m0000gn/T/turbod/fe25f7293c19b4d8/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
I ran yarn upgrade-interactive --latest to update turbo (previously I had it at 1.4.3) and ran the codemod npx @turbo/codemod migrate-env-var-dependencies which updated my turbo.json file and created a new .turbo-cookie file. Is the .turbo-cookie file necessary and should it be committed into the repository?
Gets the same warning, and the log file is empty.
On Windows 11, turbo 1.5.4
@klapec the .turbo-cookie is not necessary, it is intended to be in the directory that turbo uses for config data, not your repository. Did it appear at the root of your repo? And was it created by the codemod?
@gsoltis Thanks for the response. It did appear at the root of my monorepo, I assume it was either created by upgrading turbo to 1.5.4 (postinstall?) or by the codemod. I haven't seen exactly when was it created, I noticed it when I was about to commit my changes to the repository.
@klapec got it. we're looking into how it could've ended up in that spot, but it's safe to delete. Sorry about that!
For the windows users in this thread: it looks like we have an issue somewhere between Go, windows support for AF_UNIX, and grpc. I'm tracking down the exact fix, but I expect that's the cause of being unable to use the daemon thus far
We've published [email protected] which disables the daemon on windows while we sort out the AF_UNIX socket issue. I expect it will be re-enabled soon, as we've identified the issue and are working on a mitigation.
For those curious about the nitty-gritty details:
- The Go grpc dialer uses
url.Parseto determine how to connect to a server - Go, like many other platforms, requires that the
Pathportion of a url start with a leading/ - GRPC uses the
Pathportion of a url for the location to Dial when the network isunix/ url scheme isunix:// - This doesn't work well on windows, where file paths do not start with a leading
/ - This means that we are either sending in a url that fails to parse (no leading '/'), or a url that parses, but has an extra leading '/'.
Not to worry though, we can patch it to do special handling on windows. Coming soon.
@gsoltis Tried out [email protected] on both WSL2/Ubuntu and on the (disabled) Windows variant now. No noticeable issues - good work :)
If you have the time, could you please explain where we actually benefit from utilizing the daemon? I cannot say I notice any differences and could not find any documentation addressing that btw.
@gsoltis : after upgrading Turborepo to version 1.5.5 I still notice the same warning message on my Macbook Pro M1 (MacOS 12.5.1).
A .turbo-cookie file is created at the root of my project everytime I run a turbo command.
As already mentioned, the warning message is :
WARNING failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed. Please ensure the following:
- the process identified by the pid in the file at /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.pid is not running, and remove /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.pid
- check the logs at /Users/{user}/Library/Application Support/turborepo/logs/ba68216b5b606bf5-{project}.log
- the unix domain socket at /var/folders/33/3dflr1_x01b07gpkyb7qsxtc0000gp/T/turbod/ba68216b5b606bf5/turbod.sock has been removed
You can also run without the daemon process by passing --no-daemon
Note that I have no trouble whatsoever with my pipeline. Everything seems to work accordingly.
@gkn Absolutely, and I apologize it hasn't made it into the docs yet.
The daemon is purely a performance optimization, and not at all necessary for correct functioning of turbo. The daemon gives turbo the ability to do some work ahead of time or after a turbo run without blocking the user. The first feature that it implements today is skipping cache restoration if the outputs haven't changed since the last time a task was run. The result is faster cache hits, since they don't incur any file I/O. Some of the next features that are coming include:
- uploading to remote caching in the background so that
turbo rundoesn't need to wait for uploading - calculation of hash keys in advance so that
turbo rundoesn't need to wait to calculate what has changed, again avoiding file I/O during the time that a user is waiting
Furthermore, the daemon has an idle timeout and is disabled automatically in environments that don't "look" like dev machines: terminal is not a tty, presence of some known CI environment variables, etc.
@fveauvy can you give an example of how you are invoking turbo when you see the .turbo-cookie file show up? We are looking into it, but so far have not managed to reproduce the issue.
cc @mehulkar