All tools from shell are very slow

[mmozeiko]

Any tool I run in msys2 shell (ls, mv, ...) are very slow. Simply ls takes something like 3 seconds.

Running "strace ls" I see following output:

   15   39327 [main] ls 7460 DLL build:    2014-11-21 06:38
   16   39343 [main] ls 7460 dtable::extend: size 32, fds 0x1802FB488
  381   39724 [main] ls 7460 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   40   39764 [main] ls 7460 transport_layer_pipes::connect: Error opening the pipe (2)
   31   39795 [main] ls 7460 client_request::make_request: cygserver un-available
--- Process 7460, exception 00000000 at 000007FEFD22940D
1804450 1844245 [main] ls 7460 reg_key::build_reg: failed to create key ServicesForNFS in the registry
 2011 1846256 [ldap_init] ls 7460 cygthread::stub: thread 'ldap_init', id 0x2410, stack_ptr 0x2FCAC90
310497 2156753 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <...USERNAME...:*:1599881:1049089:....SOME_REGISTRY_KEY...:/home/...USERNAME...:/usr/bin/bash>
 6571 2163324 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Domain Users:S-1-5-21-1915207013-2615040368-3076929458-513:1049089:>
  768 2164092 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Performance Log Users:S-1-5-32-559:559:>
  402 2164494 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Users:S-1-5-32-545:545:>

Could slowness be related to this? I am on non-admin user on this machine.. Can that be avoided?

[maoueh]

Are you using LDAP to connect to your computer? If yes, this is similarly related to this issue, but I'm not 100% sure.

To test that you are facing the same issue, you could create a /etc/passwd with your account information so they are not fetch from Windows directly. This should be fairly simple to test it out, however, I never did it myself yet.

Information in this thread or on cygwin's nsswitch.conf could prove useful. Maybe others have a fairly simple cookbook to follow.

[mmozeiko]

I using Windows by logging in with domain user.

I created /etc/passwd and /etc/group files using mkgroup and mkpasswd utilities. They include all groups and usernames that were in "strace ls" output (and more, but that's irrelevant). But running "ls" is still slow. "strace ls" now show this:

   12   43607 [main] ls 14816 dtable::extend: size 32, fds 0x1802FB3E8
  236   43843 [main] ls 14816 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   39   43882 [main] ls 14816 transport_layer_pipes::connect: Error opening the pipe (2)
   21   43903 [main] ls 14816 client_request::make_request: cygserver un-available
--- Process 14816, exception 00000000 at 000007FEFD22940D
1459351 1503254 [main] ls 14816 reg_key::build_reg: failed to create key ServicesForNFS in the registry
966780 2470034 [main] ls 14816 cygheap_user::ontherange: what 2, pw 0x1802FB6B8
   41 2470075 [main] ls 14816 cygheap_user::ontherange: HOME is already in the environment /home/USERNAME
   96 2470171 [main] ls 14816 build_argv: argv[0] = 'ls'
10469 2480640 [main] ls 14816 build_argv: argc 1

As you can see it has no more "pwdgrp::fetch_account_from_windows" lines. But there are still errors:

1. something about connecting to named pipe + cygserver
2. something about failure to create registry key + ServicesForNFS

[jmichae3]

• if it's running anything over a network like a network share, you can expect significant lag
• if it's 32-bit code running on a 64-bit box, you can expect the virtualization (WOW64) to slow the program down. but hopefully not that much on a local box, especially if it's in the OS's internal disk cache in RAM. first access will be slow, subsequent accesses should be much faster due to disk cache, but the virtualization thing is another hurdle, however it doesn't slow things down too much I should think.

[KrullBorg]

my case is slightly different; for me msys is slow on starting and on bash completion; but "ls" is fast

see http://sourceforge.net/p/msys2/discussion/general/thread/87925e6f/

i also tried to create passwd/group; "mkgroup -d" doesn't output group from domain

[KrullBorg]

i "solved" disabling db in nsswitch.conf

[Alexpux]

ok.

[mmozeiko]

This is great. Commenting out "db" from nsswitch.conf helps. Now all operations are fast for me.

[legends2k]

Disabling db from both passwd and group fixed the issue but led to my username being unrecognised (bash prompt showed me as Unknown+User). However, the "fix" was to just disable db from group and leave passwd as it was, and this sped up MSYS2 utilities like diff, which, ls and also the startup of MSYS2 significantly.

[KrullBorg]

unfortunately now some commands are slower than some time ago; for example: ls, git status/diff, configure (autoreconf), gitk

[KrullBorg]

re-enabling db in nsswitch.conf makes these commands again fast; but it slows bash completion (TAB); the start of msys2_shell.bat is quite normal

[legends2k]

@KrullKorg Bash completion being slow has got nothing to do with MSYS2 itself. See here for an explanation.

[KrullBorg]

as i wrote bash completion isn't slow with db disabled in nsswitch.conf and slow if db is enabled; tomorrow i'll try again, to be sure

of course I'm talking about a pc under ms (samba) domain; at home it works perfectly

and yes i know that msys2 derived from cygwin

[KrullBorg]

i just tried under latest cygwin (1.7.5) and it works perfectly

[elieux]

Notable differences between MSYS2 and Cygwin:

• the current MSYS2 runtime is based on Cygwin trunk from April 10th, not on Cygwin release
• MSYS2 doesn't include cygserver and other Cygwin system integration tools

Can you look at differences between your Cygwin and MSYS2 installs? Namely /etc/passwd, /etc/group, /etc/nsswitch.conf, /etc/fstab, output of mount, a cygserver process running. MSYS2 should take advantage the same caching mechanisms as Cygwin, at least within a process tree.

I'm sorry I'm dumping all the debugging on you, but I don't currently have any domain to join here.

[KrullBorg]

i don't have /etc/passwd, /etc/group and cygserver in both msys2 and cygwin

FSTAB - CYGWIN

# /etc/fstab
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/using.html#mount-table

# This is default anyway:
none /cygdrive cygdrive binary,posix=0,user 0 0

FSTAB - MSYS2

# For a description of the file format, see the Users Guide
# http://cygwin.com/cygwin-ug-net/using.html#mount-table

# DO NOT REMOVE NEXT LINE. It remove cygdrive prefix from path
none / cygdrive binary,posix=0,noacl,user 0 0

NSSWITCH - CYGWIN

# /etc/nsswitch.conf
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nsswitch
#
# Defaults:
# passwd:   files db
# group:    files db
# db_home:  cygwin desc
# db_shell: cygwin desc
# db_gecos: cygwin desc

NSSWITCH - MSYS2

# Begin /etc/nsswitch.conf

passwd: files db
group: files db

db_enum: cache builtin

db_home: cygwin desc
db_shell: cygwin desc
db_gecos: cygwin desc

# End /etc/nsswitch.conf

MOUNT - CYGWIN

C:/cygwin64/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin64/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin64 on / type ntfs (binary,auto)
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
I: on /cygdrive/i type smbfs (binary,posix=0,user,noumount,auto)
K: on /cygdrive/k type ntfs (binary,posix=0,user,noumount,auto)
O: on /cygdrive/o type smbfs (binary,posix=0,user,noumount,auto)
V: on /cygdrive/v type smbfs (binary,posix=0,user,noumount,auto)
X: on /cygdrive/x type smbfs (binary,posix=0,user,noumount,auto)
Z: on /cygdrive/z type sshvfs (binary,posix=0,user,noumount,auto)

MOUNT - MSYS2

C:/msys64 on / type ntfs (binary,noacl,auto)
C:/msys64/usr/bin on /bin type ntfs (binary,noacl,auto)
C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)
I: on /i type smbfs (binary,noacl,posix=0,user,noumount,auto)
K: on /k type ntfs (binary,noacl,posix=0,user,noumount,auto)
O: on /o type smbfs (binary,noacl,posix=0,user,noumount,auto)
V: on /v type smbfs (binary,noacl,posix=0,user,noumount,auto)
X: on /x type smbfs (binary,noacl,posix=0,user,noumount,auto)
Z: on /z type sshvfs (binary,noacl,posix=0,user,noumount,auto)

PS: i just tried to remove db_enum from nsswitch.conf in msys2 PPS: don't worry about ask info; i'm happy to help msys2

[elieux]

Can you try to mount cygdrive in Cygwin with noacl? I don't see why noacl would make things slower, but it's worth a shot.

I'm assuming you're trying ls somewhere inside /c/ (/cygdrive/c/) and not inside /home/ or somewhere like that.

[KrullBorg]

i changed /etc/fstab in cygwin but i got the same behavior adding noacl: it's fast

and same behavior both in /home/ and in /c/

i also tried to remove noacl in msys2, but without improvements

[elieux]

I'm out of ideas at this point. I don't know enough about Cygwin internals to suggest a good test for you. You can try searching the Cygwin mailing list, maybe some of the information there will apply to your situation.

[KrullBorg]

i just tried on a relative new pc (virtual machine) with windows 10 tp (under the same domain) and it seems to work...

i installed msys2 from exe; then i upgraded with pacman -Syu

so i'll try to use a clean windows 8.1 vm... maybe the problem is the windows version... or my pc

[KrullBorg]

i really think it's the fault of my pc; i tried also with windows 8 and it works

i found on strace executed on my pc this error (that isn't present in virtual machines)

2388701 3040772 [main] ls 7032 pwdgrp::fetch_account_from_windows: LookupAccountSid(S-1-5-21-726227932-2052316878-829958588-513), Win32 error 1789

[elieux]

Interesting. KB976494 says the error is ERROR_TRUSTED_RELATIONSHIP_FAILURE "The trust relationship between this workstation and the primary domain failed.", so this could be a bug in Windows, or your domain connection is borked.

Can you try to leave and re-join the domain?

[KrullBorg]

yes i arrived at the same conclusion, and it worked

[k-takata]

I also faced the same issue with KrullKorg and legends2k's case. I found that expansion of ~* was very slow. I had to update /etc/passwd and /etc/group, then disable db in nsswitch.conf. More detail: https://gist.github.com/k-takata/9b8d143f0f3fef5abdab

[elieux]

Very nice summary, @k-takata.

[kjeremy]

@k-takata your workaround gist works great until I disconnect from the domain and hop onto another network... then it's slow again.

[virtuald]

I found that pacman installs are really slow until I applied @k-takata 's workaround. I wonder why it's so slow for it to do lookups on the domain?

[Globik]

And why no cygserver? Cannot run now postgresql. Error like function is not implemented.

[riban-bw]

Even after changing nsswitch.conf things are still a bit slow after dropping off an AD network. I found that disabling network and then re-enabling (using my laptop hardware button) made things responsive again.

[i2000s]

This may not be related to this issue, but Google led me here. My MSYS has been extremely slow (ran 12 hours not to finish a ./configure which could be finished in less than 1 min on Linux) using MinGW on Windows 10. All anti-virus software has been turned off, and I couldn't find the nsswitch.conf file on my installation directory. Any idea?

[legends2k]

@i2000s Are you using MSYS and not MSYS 2? They're very different! I'm pretty sure that every installation of MSYS 2 I've seen had /etc/nsswitch.conf.