Multiple tabs startup task hangs on the last tab

[hymair]

Purpose of the issue

• [x] Bug report (encountered problems/errors)
• [ ] Feature request (request for new functionality)
• [ ] Question

Version Information

Cmder v1.3.19.1181 ConEmu v21.9.12.0

Description of the issue

When starting multiple console tabs task on startup, very often the last one hangs (check screenshot).

[stale[bot]]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in a week if no further activity occurs. Thank you for your contribution(s).

[daxgames]

@hymair See latest unreleased build

I just launched Cmder 30/30 times with 4 panes as shown with no issues like you described.

[Mikaz-fr]

I observe the same issue on the new Cmder 1.3.20 with multiple tabs (no split window). Killing the process with Ctrl+K allows the init of the startup to complete. Let me know if I can provide more debug information to know where it's hanging.

[ShadowSFX]

I'm having the same issue. I keep having to restart the tab every single time i launch cmder.

[chrisant996]

If you close the non-hung tabs, does the tab stay hung?

If so, can you use Task Manager to save a dump from the CMD.exe from the hung tab, and save a dump from ConEmu.exe?

If you can either upload them here or send them to me privately at my GitHub profile address, then I could try to identify what's hanging.

Dumps can potentially contain personally identifiable data, so you may want to send privately to me. (I myself don't mind publicly uploading dumps from my machine, but I have a lot of experience with that and know what kind of my own info may or may not be in a given dump, so it's easier for me to be comfortable with that.)

[ShadowSFX]

No, the tab isn't really like "frozen". It just doesn't have anything, no prompt. I can close it without issues or restart it and it will work instantly. And it doesn't always happen, it happens whenever i start up my pc and launch cmder for the first time. I may have been misleading with "every time". I only open cmder once a day and it stays open for the whole day, so "every time" is as in "every day". If i close cmder with all of the tabs right now and re open it, the tabs will start up normally.

[chrisant996]

Can you capture the dumps for the CMD.exe from the affected tab, and for ConEmu.exe? I can take a look to see what it's stuck on or what it's doing.

[ShadowSFX]

Alright, got the dumps. How do i send them to you?

[chrisant996]

I recommend putting them in a .zip file.

Then you can upload the .zip file in a reply here, if you're comfortable with the data being public.

Or you can email to my github profile address. Or if it's too large for email, you can put the .zip file on a cloud service somewhere and send me a link to it.

If you want to be extra cautious, you could password protect the .zip file and send me the password in a private email to my github profile address.

(Personally, I would be ok with publicly sharing my cmd.exe and conemu.exe dumps, especially from the beginning of a session. But you should decide based on your own comfort level.)

[Mikaz-fr]

Thanks Chris for having a look at that issue. I'm able to reproduce also and sent you via email a couple dumps hoping it helps you to understand what is happening.

[ShadowSFX]

I just sent you the link with the 7zip file with all the dumps. Thank you!

[chrisant996]

Thanks Chris for having a look at that issue. I'm able to reproduce also and sent you via email a couple dumps hoping it helps you to understand what is happening.

Some observations about the dumps you sent:

1. Inside cmd.exe, Clink is in a normal state waiting for input. It seems to have successfully hooked all the necessary APIs in cmd.exe, and it has successfully intercepted the prompt.
   • When the tab is in that state, does typing input work?
   • Even if input is not visible, does typing something like dirEnter result in a directory listing?
   • Does typing a program name such as explorerEnter open a window?
   • UPDATE: I suspect this cmd.exe is from some other window or tab, and isn't the one that was hosting the locale.exe that got stuck.
2. ConEmu.exe seems to be in a normal idle state, waiting for input.
3. How did you realize to capture the "ActiveProcess_CEDump"? That one looks very interesting: the msys locale.exe program is stuck waiting on a handle.
   • The dump file appears to be a full heap dump (good!) but the debugger says it's unable to access the handle table, so I can't find out anything about what kind of handle (event, semaphore, file handle, etc), whether it's named or unnamed, or what its name might be.
   • If I could find symbols and source for MSYS 2.0 DLL version 3.3.6 then I could see more specifically what it's doing.
   • So far, I haven't been able to find symbols for MSYS 2.0 DLL version 3.3.6.

[chrisant996]

I just sent you the link with the 7zip file with all the dumps. Thank you!

@ShadowSFX Perfect. The "cmd(2).DMP" shows cmd.exe is waiting for locale.exe to finish.

Here is the command line, specifically, from the dump:

0:000> db 0214d1124b00h
00000214`d1124b00  22 00 43 00 3a 00 5c 00-50 00 72 00 6f 00 67 00  ".C.:.\.P.r.o.g.
00000214`d1124b10  72 00 61 00 6d 00 20 00-46 00 69 00 6c 00 65 00  r.a.m. .F.i.l.e.
00000214`d1124b20  73 00 5c 00 47 00 69 00-74 00 5c 00 75 00 73 00  s.\.G.i.t.\.u.s.
00000214`d1124b30  72 00 5c 00 62 00 69 00-6e 00 5c 00 6c 00 6f 00  r.\.b.i.n.\.l.o.
00000214`d1124b40  63 00 61 00 6c 00 65 00-2e 00 65 00 78 00 65 00  c.a.l.e...e.x.e.
00000214`d1124b50  22 00 00 00 00 00 00 00-87 45 3d 08 6e 55 00 00  "........E=.nU..

And the "locale.DMP" shows that the msys locale.exe is waiting on a handle, the same as in the dumps from @Mikaz-fr.

So, it looks like the issue is in msys_2_0.dll, and specifically in locale.exe, but I would need matching symbols and sources for msys to be able to investigate further into why msys is getting stuck.

[chrisant996]

@Mikaz-fr @ShadowSFX wow, I'm impressed that you collected dumps for more processes than I had suggested, and that the additional processes are exactly the relevant culprits (or perhaps victims 😉). Well done! That made it easy and fun to do meaningful investigation. Thank you!

[Mikaz-fr]

When the tab is in that state, does typing input work? Even if input is not visible, does typing something like dir Enter result in a directory listing? Does typing a program name such as explorer Enter open a window?

I've tried and none of that works, it's really like no input is being registered.

I also had a deeper look and could confirm the call to locale.exe -uU 2 is the one hanging. This is done part of vendor\init.bat around line 303:

for /F "delims=" %%F in ('!git_locale! -uU 2') do (
    set "LANG=%%F"
)

Since it only happens when multiple tabs are being re-opened at startup I suspect some issues when this is called on multiple threads at the same time. To confirm this I've used the following bat file:

@echo off
set localePath=<PATH>\vendor\git-for-windows\usr\bin\locale.exe
for /l %%x in (1, 1, 100) do (
    echo Loop %%x / 100
    %localePath% -uU 2
)

What's very interesting is that if I run this in multiple plain cmd.exe at the same time there is no deadlock whatsoever. But as soon as I run one at the same time from a Cmder task it makes the other hangs. Which I believe is what is happening when multiple tabs are created at the same time.

I hope this helps reproduce the issue and understand where it gets stuck. Maybe as a temporary workaround, affected users can locally hardcode in vendor\init.bat either the value in LANG to avoid calling locale.exe or add a random delay before the call to decrease the chances multiple threads collide.

[daxgames]

@Mikaz-fr if you manually create a cmd.exe task in ConEmu and do the same does it hang?

[Mikaz-fr]

@daxgames I’m not by my computer until Tuesday, maybe @ShadowSFX or somebody else that has the issue can try to reproduce.

[chrisant996]

I was able to reproduce the hang.

• Open a tab.
• Duplicate it, splitting to the right.
• Prepare both tabs by typing c:\tmp\locale_loop.cmd into them.
• In rapid succession, press Enter in one tab, then in the other.
• The first tab reached the 100th iteration and finished.
• The very instant that the first tab finished, the second tab froze (in iteration 88).

But here is the very interesting part!

• Once the second tab is frozen, go back to the first tab.
• Run "{path to git}\usr\bin\locale.exe" -uU 2Enter in the first tab.
• Each time I run that in the first tab, the locale.exe in the second tab wakes up, and then freezes again in the next iteration of the loop!

And it gets better!

• Even running "{path to git}\usr\bin\arch.exe" --help wakes up the locale.exe in the second tab!
• (And then the next iteration of the loop freezes again.)

Something seems wrong in MSYS2, indeed! Looks like some object handle is being ... well, mishandled.

Without matching symbols I can't pinpoint what in MSYS2 is wrong. I could build my own MSYS2, I suppose. But honestly that's a big investment of time and energy for me, since I've never used or built MSYS2 before. So, if someone else can build MSYS2 and verify that matching local symbols exist (.pdb file), and capture a dump of locale.exe in a hung state, and send me the dump and the msys_2_0.pdb file, then I could pinpoint what is happening inside MSYS2.

Or really, at that point, it would be possible for someone to use the gdb debugger locally to find what msys_2_0.dll is stuck on. Even just sharing a callstack with resolved symbol names would likely reveal the root cause.

---

UPDATE: Does not reproduce outside of ConEmu; but reproduces in ConEmu regardless of whether Cmder is used.

• If ConEmu already has a locale.exe that's in a hung state, then running the script outside of ConEmu (standalone cmd.exe) also runs into the problem.
• Or when I start the script outside ConEmu and then also start the script in a ConEmu tab, then the outside script completely freezes and never unfreezes, but the script in ConEmu runs to completion._

[daxgames]

@chrisant996 did you disable conemuhk injection and try?

[chrisant996]

@chrisant996 did you disable conemuhk injection and try?

@daxgames good question; I didn't. I can try that tonight.

[chrisant996]

@chrisant996 did you disable conemuhk injection and try?

• With conemuhk injection ON --> locale.exe hangs.
• With conemuhk injection OFF --> locale.exe does not hang.

[Mikaz-fr]

Thanks everybody for the efforts on this analysis. It's unclear to me what we do next though ? I have set a workaround for my local Cmder, but it's not portable and is not a fix for the issue with locale.exe.

Any idea of how to properly solve or workaround that problem ?

[daxgames]

@Mikaz-fr you could disable conemuhk injection as noted by @chrisant996 above.

This should be portable as it should be saved in the conemu settings file.

We do not develop either perts of this problem. Locale.exe is part of git-for-windows and conemuhk.dll is a part of conemu.

If I were you I would raise this question with the Conemu developer since locale.exe seems to work fine on its own.

@chrisant996 may have other recomendations though.

[chrisant996]

@Mikaz-fr yes, report the issue in the ConEmu repo, and include a link to this issue.

The little batch script you shared was very useful for reproducing the issue. Be sure to give the simple and complete repro steps when reporting the issue in the ConEmu repo -- that will make it quick and easy for Maximus to reproduce and investigate the issue.

[Mikaz-fr]

@daxgames, @chrisant996, thanks for your efforts and guidance. I've now opened a matching issue on ConEmu project (https://github.com/Maximus5/ConEmu/issues/2520) after confirming it was still present in the latest standalone ConEmu release. Feel free to follow it there and comment to give more technical details if I missed something.