Constant crashes when using remote.exe files.

[karpiyon]

Hi,

My C# app is sending the ‘RemoteControl’ exe files to openkneeboard using voice attack and a plugin I created. Those commands are not sent as admin. I usually start openkneeboard, then run the game and then it usauly crash. I then restart openkneeboard and it may work for some time.

I am running Falson Bms using steamVR, win 10, nvidia 1080ti. Everything is updated.

I doubt the plugin is not causing this since I tested the functionality many times when the game was not running, meaning without VR and it never crashed.

I'll send another dump file and if it is not enough I guess I can produce a larger one since the issue will surly happen again. My dump file

Dan

[fredemmott]

unfortunately, the minidump contains no useful information ; can you attach a full dump?

[fredemmott]

Is there a specific remote exe that triggers this?

[karpiyon]

Not sure. Mostly I get it when I enter the game and call my “phrases” tab though it also happens with other tabs as well. But I think it happens when I start it, let it run for a while and then start the and show it.

The way I do it, all via remote is this:

1. Update the tab content, in this case the pdf the tab is pointing to
2. Call SHOW.exe
3. Wait 200ms
4. Call SET_TAB.exe name tab_name

It does not crash when not in VR. I added 200ms wait because in I send e.g.: SHOW.exe, flowed by NEXT_TAB.exe, it will always jump 2 tabs. I found that adding some delay makes it work as expected.

From: Fred Emmott @.> Sent: Monday, July 8, 2024 13:54 To: OpenKneeboard/OpenKneeboard @.> Cc: karpiyon @.>; Author @.> Subject: Re: [OpenKneeboard/OpenKneeboard] Constant crashes when using remote.exe files. (Issue #609)

Is there a specific remote exe that triggers this?

— Reply to this email directly, view it on GitHubhttps://github.com/OpenKneeboard/OpenKneeboard/issues/609#issuecomment-2213681016, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHCZEKPVA4Q4C73O24QBJDTZLJVUZAVCNFSM6AAAAABKQA7WE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJTGY4DCMBRGY. You are receiving this because you authored the thread.Message ID: @.@.>>

[karpiyon]

I will attempt a full dump

From: Fred Emmott @.> Sent: Monday, July 8, 2024 13:51 To: OpenKneeboard/OpenKneeboard @.> Cc: karpiyon @.>; Author @.> Subject: Re: [OpenKneeboard/OpenKneeboard] Constant crashes when using remote.exe files. (Issue #609)

unfortunately, the minidump contains no useful information ; can you attach a full dump?

— Reply to this email directly, view it on GitHubhttps://github.com/OpenKneeboard/OpenKneeboard/issues/609#issuecomment-2213674055, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHCZEKMJH55FTLYFJMMBGI3ZLJVH7AVCNFSM6AAAAABKQA7WE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJTGY3TIMBVGU. You are receiving this because you authored the thread.Message ID: @.@.>>

[fredemmott]

SHOW.exe, flowed by NEXT_TAB.exe, it will always jump 2 tabs.

Are you able to reproduce this outside of your plugin? If so, could you file a separate issue with reproduction steps?

[karpiyon]

I'll try, Are the dumps I emailed you any good, or should I produce the full ones?

[fredemmott]

Not able to access my email now, can check later; if they are minidumps rather than full dumps, they are most likely not useful, assuming the crash is the same as the one you previously attached.

[karpiyon]

Hi,

I am trying different sequences to see if I can understand when what causes the crashes. What should be the min delay between 1 remote command to another? Perhaps this is what is causing the crashes? Or perhaps it has something to do with the fact that I am updating a tab which is selected?

If I want to update an existing tab, that is rewrite the pdf it is showing, should I simple update the pdf, when the tab is visible or should I first: Hide OK ==> updated the pdf ==> show OK Or perhaps: Hide OK ==> select another tab ==> updated the pdf ==> select the tab ==> show OK, I think the latter produced less crashes for me.

I noticed that is I send SHOW, Followed by Next_TAB it will skip 2 tabs, as you mentioned above, however. Does it make sense then, that if I send the same sequence but wait 200ms in between, I get the real NEXT tab and not the next-next tab?

From: Fred Emmott @.> Sent: Monday, July 8, 2024 15:49 To: OpenKneeboard/OpenKneeboard @.> Cc: karpiyon @.>; Author @.> Subject: Re: [OpenKneeboard/OpenKneeboard] Constant crashes when using remote.exe files. (Issue #609)

Not able to access my email now, can check later; if they are minidumps rather than full dumps, they are most likely not useful, assuming the crash is the same as the one you previously attached.

— Reply to this email directly, view it on GitHubhttps://github.com/OpenKneeboard/OpenKneeboard/issues/609#issuecomment-2213987676, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHCZEKPLZMIUCTESWZE4NSTZLKDD7AVCNFSM6AAAAABKQA7WE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJTHE4DONRXGY. You are receiving this because you authored the thread.Message ID: @.@.>>

[fredemmott]

What should be the min delay between 1 remote command to another?

There is no minimum delay; they are tested running continuously in a loop, i.e. microseconds apart.

The mini dumps you sent are unfortunately not useful; unable to investigate this without either a way to reproduce the problem or full dumps.

[karpiyon]

Can the command sequence make any difference as I suggested in the last reply?

If there is no minimum delay, what can explain that SHOW ==> NEXT_TAB will show jump 2 tabs ahead, while SHOW ==> wait 200ms ==> NEXT_TAB. Only jumps ahead 1 tab?

From: Fred Emmott @.> Sent: Monday, July 8, 2024 21:01 To: OpenKneeboard/OpenKneeboard @.> Cc: karpiyon @.>; Author @.> Subject: Re: [OpenKneeboard/OpenKneeboard] Constant crashes when using remote.exe files. (Issue #609)

What should be the min delay between 1 remote command to another?

There is no minimum delay; they are tested running continuously in a loop, i.e. microseconds apart.

The mini dumps you sent are unfortunately not useful; unable to investigate this without either a way to reproduce the problem or full dumps.

— Reply to this email directly, view it on GitHubhttps://github.com/OpenKneeboard/OpenKneeboard/issues/609#issuecomment-2214843448, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHCZEKK6ZYZBBOHE4APPDK3ZLLHW3AVCNFSM6AAAAABKQA7WE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUHA2DGNBUHA. You are receiving this because you authored the thread.Message ID: @.@.>>

[fredemmott]

show should not affect next tab; most likely the issue is calling next tab before the file has fully flushed to disc, with the show call being irrelevant.

To answer more concretely requires:

Are you able to reproduce this outside of your plugin? If so, could you file a separate issue with reproduction steps?

[karpiyon]

Hi, I modified all my calls to explorer.exe <remote.exe> into inline calls to OpenKneeboard_CAPI64.dll I sill get crashes all the time. What I am doing is simply updating the pdf which and showing the tab the pdf is linked to. I write to teh pdf:

                    using (var stream = new FileStream(outFile, FileMode.Create))
                    {
                        var pdfDoc = new Document(PageSize.A4, 10f, 20f, 20f, 10f);
                        PdfWriter.GetInstance(pdfDoc, stream);
                        pdfDoc.Open();
                        pdfDoc.Add(pdfTable);
                        pdfDoc.Close();
                    }

(The Close makes sure it is flushed as well) I then send an OpenKneeboard_send_utf8 command to select the TAB followed by an OpenKneeboard_send_utf8 command to show OK regardless if it is shown or not. The scenario is similar to teh one before where I use to send the same sequence via the remote commands. The are 2 dump files: 1 s roughly 133M zipped (590M unzipped) and the other 895MB zipped (8.3G unzipped)

Link to dump folder

[fredemmott]

Thanks; unclear if PDF is actually 0 pages, or if it's attempting to render before it's reloaded (after the file is modified). Either way, obviously it shouldn't crash.

[fredemmott]

Not getting a crash, but getting deadlocks and various other issues with the quick start PDF added as a tab; in powershell:

$file = Get-Item 'C:\Users\fred\code\OpenKneeboard\docs\Quick Start.pdf'
.\OpenKneeboard-RemoteControl-HIDE.exe; .\OpenKneeboard-RemoteControl-RELOAD_CURRENT_TAB.exe; .\OpenKneeboard-RemoteControl-SHOW.exe

Calling 'HIDE' is optional but make the repro more reliable. This generally results in a deadlock on a shared mutex.

[fredemmott]

Just spamming $file.LastWriteTime=(Get-Date) is enough to cause some issues. Might not be the same root cause, but there's definitely some buginess that needs fixing in anything happening shortly after a PDF file modification.

The dumps you shared have:

• PDF objects with no pages
• after checking if there are pages, attempting to render a specific page
• .... but the PDF object has no pages
• ... ultimately tries to create an 0x0px texture

[karpiyon]

What workaround do you suggest I implement in the meanwhile? hide OK > select another page > update pdf > select the pdf link page > show OK if yes, in which order? Do you need more data? Also, does the fact that the same process does not crash without VR makes sence?

[fredemmott]

What workaround do you suggest I implement in the meanwhile?

Update PDF, then wait some amount of time before doing anything else with OpenKneeboard

You should not hide/show OpenKneeboard unless explicitly asked to by the user, and it will not solve this problem as the app will still render.

Do you need more data?

No, thanks.

Also, does the fact that the same process does not crash without VR makes sence?

This is somewhere between 'coincidence' and 'race condition frequency may be higher when system load is high'

[fredemmott]

My freeze appears related to your crash:

• In release builds, it crashes
• In debug builds, the D3D11 debug layer raises an error for attempting to create a 0x0px texture, says it will trigger a breakpoint, but doesn't ; the same exceptio nis thrown, but it does not terminate the process, just the thread
• So, the D3D11 subsystem crashes OpenKneeboard in release builds, but in debug builds, it just termiantes a thread that held a mutex open, leading to a freeze

[karpiyon]

Thanks, I'll try this:

            FileIo.createPDF(filteredDataTable, "phrases.pdf");
            Thread.Sleep(200);//is 200ms enough?
            OK.SendCommand("show");
            OK.SelectProfileDayNight(); //This selects Day/Night Profile if the user selected these option in the plugin configuration.
            OK.SendCommand("selectTabByName", "JanJan Phrases");//Select the tab linked to "phrases.pdf", created above

[fredemmott]

This is another bug caused by https://github.com/microsoft/WindowsAppSDK/issues/3506 - OpenKneeboard's PDF viewer is an inconsistent state due to that bug - it makes it so that multiple non-async functions are in progress in the same thread at the same time without calling each other.

[fredemmott]

I'm still able to make this crash, but only in extreme cases, e.g. replacing the file 1000 times per second. At that point, I'm hitting more bugs in Windows.Data.Pdf which I can't workaround. Will do some more varied tests then close this.

[fredemmott]

With that last change, I can usually modify the file 1000 times at 1ms intervals without issue; when there is a crash, it's internal to Windows.Data.PDF and only Microsoft can fix it.

[fredemmott]

These changes are too invasive to pull into v1.8; until v1.9 or above, you'll want to stick with 'sleep a bit after modifying the PDF before doing anything else"

[karpiyon]

I tried the following and it still crashes: Instead of creating the pdf OK is linking to I do:

• Create phrase.tmp.pdf
• Copy phrase.tmp.pdf to phrase.pdf
• Wait 250ms
• Show OK
• Select the tab which is linked to the pdf.

I guess it's the same but since it keeps crashing I am supplying the new dump. New dump here: link to dump filder

Any workaround you can think of which would help?

[fredemmott]

Thanks, that dump shows the same problem; further dumps on that version are probably not useful.

Copying over is going to have the same problem as overwriting.

There is no specific 'good' value; in my case, it's 1ms, but it depends on system, load, and just plain bad luck - that MS bug can cause a crash even after an infinite wait.

Can you test with the version at https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/9901144922 ? Only use that version to test this issue - it has other problems, and is not generally suitable for use. DO NOT TELL YOUR USERS TO INSTALL THIS VERSION.

If it's fixed there, it will be fixed in the next feature release of OpenKneeboard (which has no ETA). Otherwise, please attach a new dump.

As for workarounds, if you can't find a reliable-enough time to wait, then you need to either use a different file type (e.g. png, txt), a different tab type (e.g. window capture), or properly create your own overlay (https://openkneeboard.com/faq/third-party-developers.html#how-do-i-use-openkneeboard-to-create-my-own-openxr-overlay)

[karpiyon]

Thanks. I'll test and let you now. PNG sound like a good solution for now.

[karpiyon]

I run a quick test with the new version you supplied and so far no crashes. I'll keep testing and let you know if it does not crash anymore.

[fredemmott]

Thanks; I’ll take another look at how practical it is to bring some of these fixes into v1.8, but not expecting to get them all.

[fredemmott]

Wasn't as bad as I expected - could you try https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/9911651289 ? If that works well, I can put the fixes into the next bugfix release.

Again, that version is for testing only, and is otherwise unsupported - do not tell your users to use that version.

[karpiyon]

I can’t install it. I get a message that a later new version is already installed. The previous debug version you sent me was OpenKneeboard-v1.9.0.2165.msi, this one is OpenKneeboard-v1.8.5.2166. Shall I uninstall first? Will it keep all my settings?

From: Fred Emmott @.> Sent: Saturday, July 13, 2024 01:19 To: OpenKneeboard/OpenKneeboard @.> Cc: karpiyon @.>; Author @.> Subject: Re: [OpenKneeboard/OpenKneeboard] Crash when attempting to render PDF while it's in the process of reloading (Issue #609)

Wasn't as bad as I expected - could you try https://github.com/OpenKneeboard/OpenKneeboard/actions/runs/9911651289 ? If that works well, I can put the fixes into the next bugfix release.

Again, that version is for testing only, and is otherwise unsupported - do not tell your users to use that version.

— Reply to this email directly, view it on GitHubhttps://github.com/OpenKneeboard/OpenKneeboard/issues/609#issuecomment-2226428115, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHCZEKMDFH5KHPZRVREGQB3ZMBI4PAVCNFSM6AAAAABKQA7WE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRWGQZDQMJRGU. You are receiving this because you authored the thread.Message ID: @.@.>>