Call of Duty: Black Ops II [BLUS31011] Fatal Device Lost Error

[Darkhost1999]

Quick summary

Note this issue is resolved/bypassed by using Approximate Zcull Accuracy. The wiki does mention improved performance with Approximate Zcull Accuracy. https://wiki.rpcs3.net/index.php?title=Call_of_Duty%3A_Black_Ops_2 However, The use of accuracy settings should not cause a fatal error

Details

When booting BLUS31011 updated to 1.19 the game crashes at the press start screen if using 100% default configuration. But by changing zcull to approximate or relaxed we can actually play the game. This same fatal error and device lost affects so many of my games that I am glad to have obtained this game and am able to make a replicable debug output log.

·F 0:00:58.200436 {RSX [0x80558bc]} SIG: Thread terminated due to fatal error: Assertion Failed! Vulkan API call failed with unrecoverable error: Device lost (Driver crashed with unspecified error or stopped responding and recovered) (VK_ERROR_DEVICE_LOST)
(in file C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\rpcs3\Emu\RSX\VK\VKQueryPool.cpp:45[:4], in function poke_query)
(in file C:\Users\ContainerAdministrator\AppData\Local\Temp\cirrus-ci-build\rpcs3\Emu\RSX\VK\vkutils\shared.cpp:107[:24], in function die_with_error)

Attach a log file

Debug.log Crash.log Working-Approximate Zcull Accuracy.log

Attach capture files for visual issues

Not a visual issue

System configuration

Ryzen 5 3600 Gtx 1660S Windows 11 beta insider

Other details

I hope this helps end the device lost errors. I know there is a workaround but it's not a proper fix

[kd-11]

If it's consistent on your system you can install SDK and capture a log with debug output enabled. Also missing steps to reproduce, it's a large game, I don't know if it crashes in the menus or a specific level, etc.

[Darkhost1999]

If it's consistent on your system you can install SDK and capture a log with debug output enabled.

Debug.log

Also missing steps to reproduce, it's a large game, I don't know if it crashes in the menus or a specific level, etc.

When booting BLUS31011 updated to 1.19 the game crashes at the press start screen if using 100% default configuration.

[kd-11]

I'm unable to reproduce on RTX 3050. The debug output also has no information about any violation that could cause a crash. There is some log spam about use of barycentrics, but that shouldn't crash and has nothing to do with zcull accuracy. You can easily prove that this is not the culprit by setting shader quality to low.

[kd-11]

I have an idea - try this build: https://cirrus-ci.com/task/5489746785337344 Let's see if it solves the nvidia silent crash problem.

[Darkhost1999]

There was a change. The fatal error is now before the intro cutscene rather than after. Debug.log RPCS3.log

[kd-11]

We just have to wait for nvidia to implement VK_EXT_device_fault then. AMD already ships that extension, basically whenever the driver crashes it should give some more information to us about why it did so. Until then I think it's not very useful to keep fighting the NVIDIA driver.

[kd-11]

Try with https://github.com/RPCS3/rpcs3/pull/13875

[Darkhost1999]

This is using https://github.com/RPCS3/rpcs3/pull/13875/commits/bc9633396e35180563215c4341f0374f392b158b RPCS3.log -Edit add debug output log- RPCS3.log

[Shoegzer]

Getting something similar here, though not quite the same. The game doesn't crash at the start screen on default settings, but rather freezes ingame after a while (between 3-15 minutes), usually in the "zombies" mode. Using default settings except for Approximate ZCull accuracy and Ordered&Atomic RSX FIFO accuracy. Note though, the "device lost" message isn't thrown to the log file. The log below is quite large as it took about 15 minutes of play to trigger the freeze.

OS: Linux Mint 21.1 / kernel 5.15 CPU: Intel i7-10870H GPU: Nvidia GTX 3070 / driver 530.41 Commit tested: 1371bf89e04eedb4879ce97562af4f07f392f45d Log: RPCS3.log.zip

[Shoegzer]

Update: Gave this a closer look. As I mentioned I don't see the VK_ERROR_DEVICE_LOST exception in the log, though I do see CELL_NET_CTL_ERROR_NOT_CONNECTED as @Darkhost1999 has reported:

19386, spu:0]; [soft:665 hard:727]; Switches:[vol:187178 unvol:14150]; Wait:[9.577s, spur:131010]
·E 0:21:02.668006 {PPU[0x1000000] main_thread} SYS: Stat: 'cellNetCtlGetInfo' failed with 0x80130108 : CELL_NET_CTL_ERROR_NOT_CONNECTED [x2012]

This is soon followed by other entries that seem to reflect a problem, for example:

·W 0:21:02.668038 {PPU[0x100000a] Thread (hang detect) [0x0033d844]} PPU: 'sys_timer_usleep' aborted (9.539612s)

Failed attempts to resolve include:

• SPU Decoder: ASMJIT
• Framelimit: 30
• Sleep timers accuracy: [all settings]
• Driver wake-up delay: 200us
• RSX FIFO accuracy: Atomic&Ordered
• Accurate PPU 128 Reservations: Always Enabled
• PPU Thread Count: 1

Note that I cannot test OpenGL as rpcs3 hangs during the initial shader build process (though this happens with all titles, not just this one).

[Darkhost1999]

Issue not occurring on RPCS3 v0.0.29-15463-edf4f7ea Alpha RPCS3.log

[Shoegzer]

Still freezing here unfortunately, after several single-player zombie mode games. Tested against commit edf4f7eacc4d1c87fc1fade32f1662e49e2748ad with new cache and default configuration as well. Please re-open.

·E 0:25:41.404741 {Emulation Join Thread} SIG: Thread [rsx::thread] is too sleepy. Waiting for it 4226396.399µs already!

Log: RPCS3.log.zip

[Darkhost1999]

Looks like when I tried today. Reinstall your game update files and try again. https://wiki.rpcs3.net/index.php?title=Help:Installing_Game_Updates#Removing_game_updates/game_data

[Shoegzer]

Thanks for reopening. I tried what you suggested but it still froze. Update path was from 1.00 > 1.09 > 1.19, was that yours?

Log: RPCS3.log.zip

[Asinin3]

Test with https://github.com/RPCS3/rpcs3/pull/14657 (if its merged when you're reading this, just test latest RPCS3 build instead)

[Shoegzer]

Thanks, still freezes after 5-15 minutes after going ingame in testing a build with #14657. Again I do not see VK_ERROR_DEVICE_LOST in the logs but rather the others as I noted above.

New log in case it helps: RPCS3.log.zip

[Darkhost1999]

Yea freezes for me as well. I don't obtain freezes when zcull accuracy is changed from precise.

[Shoegzer]

Well I can certainly create a new issue for this, as this one's title may be misleading assuming the fatal device error isn't a cause of freezing any more.

With regards to zcull accuracy, I've tested using precise and approximate and freezing occurs with either (though perhaps not as frequent with approximate). Play in zombie survival mode, specifically starting in the bus station, and you should see the same thing?

[kd-11]

Does the driver still crash @Darkhost1999 ?

[Darkhost1999]

RPCS3.log.gz On RPCS3 v0.0.33-16999-938306a7 I can not finish the first mission when using precise zcull. My old comment I assume holds true

I don't obtain freezes when Zcull Accuracy is changed from precise.

[kd-11]

Well, the log doesn't show any RSX crashing though. The only difference between precise and approximate is that precise makes the RSX code run slowly through the same steps. I've updated the title to reflect that this issue is not causing a device lost error anymore.

[Shoegzer]

Freezing still occurs with ZCULL accuracy set to approximate, unfortunately. Other settings for this test: RSX FIFO accuracy was set to Atomic and Driver Wake-Up Delay at 280. Tested against the latest commit 6bb01066fba5cf2b569d31b4ddad7caf235bfaa7. I don't believe there's any way to prevent this title from freezing, I've tried many different settings.

Here's a log from it freezing after about 30 minutes of play in zombie mode, single player: RPCS3.log.zip

[kd-11]

Thanks for confirming. In that case, this likely has nothing to do with RSX anymore.

[Shoegzer]

Sure thing. Looking at the logs you'll see the same CELL_NET_CTL_ERROR_NOT_CONNECTED followed by Thread (hang detect) [0x0033d844]} PPU: 'sys_timer_usleep' aborted exceptions being thrown as noted above from older builds. I'm not sure if that's a clue as to the root cause, it may simply be a benign message about not connecting to network services, but it looks possible to me at least.

[Shoegzer]

Update: Actually scratch that, CELL_NET_CTL_ERROR_NOT_CONNECTED is probably a red herring as it doesn't seem to be an unintended error but rather signifies an inability to get information about the connecting system since network services aren't turned on. Would make sense anyway.

I've adjusted just about every setting exposed to at least reduce freezing occurrences (same for several others like #16393 above and #16087), but all these titles eventually freeze regardless.

[Darkhost1999]

I have settings I've used for the Assassin's Creed series to almost remove all freezing. The settings were

• PPU Interpreter (cpu tab)
• Disable PPU non-java mode fixup (advanced tab)
• Enable PPU non-java mode (debug tab)
• Ordered and Atomic RSX accuracy (advanced tab)
• Possibly Accurate RSX Reservation Access (advanced tab)

There were changes to Accurate RSX Reservation Access since my last full test. And these Call of Duty Black ops games display different reasons for freezing than the games these settings have been tested with. Still worth trying any combination of these settings if desired.

[Shoegzer]

Thanks @Darkhost1999, that's helpful and I'll try those combinations with some of the other titles. Which AC titles exhibited freezing in your testing? I ask because I tested AC1 to completion and it never froze.

It's also important to distinguish between freezing and crashing. The titles above all freeze, while others (such as in #16452) cause RPCS3 to crash entirely. I don't know if your settings would help with crashing issues but I can try them anyway.

[Darkhost1999]

Which AC titles exhibited freezing in your testing?

Brotherhood and beyond. So from the middle of the Ezio trilogy to every single game after that released on PS3. 1 and 2 unharmed. They exhibit behavior as if pausing while music keeps running in the background

[Shoegzer]

That's exactly what happens with #16087 (music playing, gameplay frozen), I haven't checked logs for comparative indicators but I'll try the settings there anyway.

Also, realizing we're going OT but if you have a chance to document your settings in relevant entries like here I'm sure that would help others in the future.

[Darkhost1999]

Also, realizing we're going OT but if you have a chance to document your settings in relevant entries like here I'm sure that would help others in the future.

https://github.com/RPCS3/rpcs3/issues/12807#issuecomment-1717341201