ValveSoftware
[PREVIEW] [BIOS] F7A0116 Breaks Sleep/Wake functionality - Board Rev. 6 F7A0113 Works
Your system information
- Steam client version: Preview
- SteamOS version: 3.5 (Preview)
- Opted into Steam client beta?: [Yes/No] Yes
- Opted into SteamOS beta?: [Yes/No] Yes (Preview)
- Have you checked for updates in Settings > System?: [Yes/No] Yes
Please describe your issue in as much detail as possible:
After 3.5 got pushed to Preview channel, I wanted to test the new features. My Steam Deck was running SteamOS 3.4.11 with bios FA70110.
After upgrading, SteamOS went to 3.5 and bios version to F7A0116. There I started seeing the issues with sleep/wake described below.
I tried returning to Stable channel and SteamOS version went to 3.4.11 but bios version stayed at F7A0116. In Stable channel I also had the same issues with sleep and wake after returning.
I re-flashed the bios included with the Stable channel package with
sudo /usr/share/jupiter_bios_updater/h2offt /usr/share/jupiter_bios/F7A0110_sign.fd -all
and the issue went away.
Since SteamOS does not always flash the bios when switching the channels, I've been able to test combinations of bios and SteamOS versions
| BIOS Version | SteamOS Version | Sleep/Wake works |
|---|---|---|
| F7A0110 | 3.4.11 | Yes |
| F7A0110 | 3.5 | Yes |
| F7A0116 | 3.4.11 | No |
| F7A0116 | 3.5 | No |
Based on these results, the issue seems to be related to the bios version. If additional logs/debugging information is needed, please let me know how to generate them.
Thanks!
Steps for reproducing this issue:
- Upgrade to SteamOS 3.5 (preview)
- Ensure latest bios is applied (FA7A0116)
- Press power button in game mode (sleep)
- Wait for a couple of seconds and press power button again (wake)
Expected Result: Steam deck beeps, wakes from sleep and displays previous state. Allows for continued use
Actual Result: Steam deck beeps. Screen is not powered on. Trackpad haptics work. No sound from previous running application
I've done some additional testing. With bios 116, journalctl doesn't even report the "wake" event in system logs. SteamOS 3.5 seems to have changed the way sleep is handled from 3.4.x.
These log files have been generated from exactly the same OS build, the only difference is the bios version that has been manually installed for each test.
I've dumped the logs with bios 110 and 116 and extracted them with:
journalctl --boot=-1 --reversed # for bios 116, as the deck was KO after a sleep
journalctl --reversed # for bios 110
For bios 110, the "wake" event happens around line 730
I'm not very experienced on debugging this kind of things, any help will be appreciated. From the looks of it, the kernel is not even receiving any event when trying to wake up...
Thanks!
UPDATE: I've been testing with the bios revisions from https://gitlab.com/evlaV/jupiter-PKGBUILD with all the versions between 0110 to 0116 installing them with sudo /usr/share/jupiter_bios_updater/h2offt <file>:
- 0110 -> Wake up works
- 0113 -> Wake up works
- 0115 -> Wake up DOESN'T work
- 0116 -> Wake up DOESN'T work
So, it seems the issue has been introduced in a change between 0113 and 0115.
Checked my Board revision in bios and it says Version "6".
Any idea? Thanks!
UPDATE2: Tried a reinstall of SteamOS using recovery image and a same output
Hey @rbordallo12 , thanks for the report. I tried your steps here (with FW 0116) and couldn't reproduce. In my Deck, suspend/wake works fine, I've tried doing that from Gamescope, and from 2 games (Brotato and Cuphead).
It's interesting (from your log) that after the message kernel: PM: suspend entry (deep) nothing is observed...
Can you paste the output of these 2 commands here?
uname -r
grep "BUILD\|VERSION_ID" /etc/os-release
Also, some questions: is your Deck plugged in some Dock station or connected to some external monitor? What game exactly were you running during the test? Are you using some form of external storage, like SD card or USB drive?
Thanks in advance!
Hi @guilhermepiccoli , here's the output for both commands:
(deck@steamdeck ~)$ uname -r
6.1.52-valve2-1-neptune-61
(deck@steamdeck ~)$ grep "BUILD\|VERSION_ID" /etc/os-release
VERSION_ID=3.5
BUILD_ID=20230915.100
(deck@steamdeck ~)$
I've reinstalled SteamOS using the recovery image and switched to Preview again. Results are the same.
From the logs, it looks like the system is not receiving any "event" when waking up. The reason i think this is bios related is because I've tested with all available versions between 0110 to 0116 and looks like the "breaking" change was introduced in 0115.
On logs, for 110 version, after entering suspend I see a line Filesystems sync:...that is not present in 116 logs
Thanks!
Thanks for your prompt answer. So, about the Dock station / external monitor / and the game your running while testing, could you give me details?
Also, if you could attach the output of sudo dmidecode here (as a file), would be nice to compare our Decks - mine is running exactly the same image than yours, so that makes me think either Dock/external monitor/game or a HW difference.
Cheers!
Hi @guilhermepiccoli thanks for your help on this issue. This is driving me crazy To be honest, I don't think the external monitor is making any difference here. I've reproduced the issue in battery mode without any usb peripheral attached. Here are the logs for bios 113 (wake works) and 116. Scenario for both logs is power up from shutdown state, reach game mode landing page and press power button to enter sleep. Then wait about a minute and press power again
Also, the dmidecode output as requested dmidecode.txt
Thank you!
Yeah, a apart of serial numbers and the DRAM model, our HW seems identical.
Can I ask you one more test?
a) Power-off the device and unplug the power cord (battery > 50% just for consistency). b) unplug any external monitors, SD card, any USB attached device and boot the Deck. c) Try the suspend procedure using FW116 from the Steam menu (Gamescope).
Lemme know the result of this experiment - no need for log collection, just your "yes/no" if that works or if reproduces the issue. If that fails, we could start trying some kernel options next week to dump status of the suspend process, in order to see where it breaks. Already anticipating that: I understand you have SSH access to the device, right?
Thanks again for the informative report and all the prompt tests =)
Hi @guilhermepiccoli performed your test:
- Battery at 75%
- Nothing plugged (not even sd card)
- Suspending from menu (by pressing power button for a couple of seconds)
Exactly the same result. System suspends correctly (I see animation) but on wake blank screen, trackpad haptics and slight fan noise.
I don't have ssh access to the device, I've been extracting the logs directly from the device itself, I'd need to set it up (let me know if there's some doc i can follow for that)
Thank you!
Hi @guilhermepiccoli , i've continued testing and I'm pretty certain that the root cause for this is some BIOS issue.
I've re-imaged my Steam deck using latest recovery image. After re-imaging, I tested with current stable version and 116 BIOS and issue was there. Switched to preview and same, issue still there.
I've also tested with an external Drive with a winndows to go install. This worked fine on 110 BIOS but on 116 the issue is exactly the same.
I've also observed that if after the failed wake, i press power button again, It seems like It goes to sleep again (no haptic feedback) and pressintg It again will return the deck to this weird state (haptic feedback)
I had this. Same issue when installing with F7A0113. It fixed itself after a couple of ~~days~~ hours. I have no clue why. I have a report ~~3~~ 7 months ago for the same exact problem. Try to put the deck into Battery Storage mode and wait a while.
Hi @RodoMa92 @guilhermepiccoli , seems like my device had a HW problem after all...
Today I wanted to test a little further and decided to re-image my device with the recovery image. After install, I re-tested the sleep/wake issue and it was still present.
As seen online, I tested if the issue could be solved by putting the device into battery storage mode. After putting it through bios and powering itself off, I put the charger on to boot up the device.
After booting, something struck me weird that i was not seeing the Battery icon in game mode. I checked on the quick access menu and no battery info was there. I rebooted the device through the menu and mid boot (Steam logo was shown) the device powered itself off.
I saw a small smoke cloud coming from the vent. I tried to boot the device again but it was completely dead. Also, when trying this, I saw another smoke cloud coming from the vent. At this point I decided to open up the device to disconnect the battery to prevent any fire.
I've opened a support ticket to get my deck RMA'd 😢
Seems like my device had some latent HW defect or something and the upgraded bios just uncovered it 😞
Thank you for your help.
Replying to https://github.com/ValveSoftware/SteamOS/issues/1138#issuecomment-1722452726
I'm sorry for your hardware issue. Seems that your battery charge controller has gone up in smoke, which is sadly the weaker part of this device.
I hope that you will have a quick RMA at this point.
Marco.
Wow @rbordallo12 , quite an odd issue! So in the end it seems really a HW fault, let us know once the RMA process is done if the new HW sleeps/wakes fine, as expected.
When I test sleep I notice on wake performance quite bad than it was on stable. It lasts for like 3 minutes then it increases but has a unstable frame rate. It used to not do this. Might be a different issue or the Same.
Had this same issue after upgrading to 3.5. Fortunately the issue was gone after going back to 3.4.10.
Hi guys, I updated on channel "Main" which is on 3.6, and it force updated the BIOS to F7A0118, which stops smokeless and sdunlock and all AMD menus from being unlocked. Can I politely ask, how do you download and go back to BIOS 116 instead? I can't seem to do that and I know nothing about linux.
When I tried the command way above, substituting the name: sudo /usr/share/jupiter_bios_updater/h2offt /usr/share/jupiter_bios/F7A0110_sign.fd -all and I changed f7a0110 to F7A0116, it said file could not be found.
Please help.
Edit I found out how to go back to bios F7a0116. had to go back to beta candidate which has 116 in its package, and OS 0915.1000 which had BIOS 116, rather than 0918 which was using bios 118, then I was able to run the command " sudo /usr/share/jupiter_bios_updater/h2offt /usr/share/jupiter_bios/F7A0116_sign.fd -all
Which then flashed the included BIOS and now I'm back on 116.
There is a third party repository with all the BIOSes released by valve, if you search around.
@guilhermepiccoli Can confirm the issue for my Steamdeck: exact same symptoms for F7A0116 (no wake from sleep) with Board Revision 6, but after downgrade to F7A0113 everything is working as expected.
I can add: after the Update (3.4.10->3.5, 113 -> 116) the "incomplete" wakeup described in the issue occurs only when plugged into a power source, but not on battery power.
Same black screen issue on wake for me going from 3.4.10 to 3.5 and BIOS 116. Also rev 6 board. I’m also seeing loss of sound when wake does work. This persists on reboot, but not power off and power on, which does restore sound. Unsure if it’s related. I’ve since downgraded the BIOS to 110 and version to 3.4.10 and that seems to have reverted everything to proper behavior
Can confirm exact same issue here. Will try downgrading to 110/113 and check. Hopefully my deck does not go up in smoke.
EDIT: Downgraded to 110 and all seems to be working fine. Will keep this issue in mind to see if future BIOS versions fix this.
Hey, if you are experiencing this issue we'd like to get some more info from your units.
Could you go to Settings->System scroll to the bottom and submit a System Report to steam support. Then send me your steam account ids or profile link.
Hey @lostgoat just did so steamid jawkuz
edit: after I sent the report i reverted back to fw113 and can confirm my SD's wake from sleep function works again.
I'd like to hop on this bandwagon. Downgraded to F7A0110 and it seems to work as expected now. Was also on F7A0116 and sleep mode wasn't working as expected. Haven't tried F7A0113 yet.
Fan revision Taicang part ending with NOOP, so I guess I have an early unit.
For those running with with SteamOS 3.5+ and BIOS 119, if any of you have SSH access, could you try doing an experiment for me?
Be sure there is no USB device connected (not even the power supply)...then ssh to the Deck and run, as root user:
echo mem > /sys/power/state
I expect the Deck will sleep. After ~30 seconds of sleeping, just try to wakeup and lemme know if that works or if it still can't wakeup.
Notice that, during this test would be good to stay connected to the Deck through SSH - I've seen cases of flaws in suspend/resume that rendered the GPU broke but all the rest was working. If that's the case, we can investigate more since it's possible to collect a dmesg using the ssh session.
Thanks in advance!
Gave it a shot - my session times out though after sending the command and restarting. Subsequent attempts to ssh into the deck do not work unfortunately until a hard reset is initiated (holding the power button).
Also checking previous boots (using journalctl --list-boots) - the session where the failed wake from sleep is missing.
For reference:
IDX BOOT ID FIRST ENTRY LAST ENTRY
-5 8551a08a32a24048aa445fad8b327188 Sat 2023-12-02 18:35:28 CET Sat 2023-12-02 18:44:06 CET
-4 7898f361c03749e1964e47601c7702ed Sat 2023-12-02 18:47:11 CET Sat 2023-12-02 19:02:26 CET
-3 6c48fccdcac0489e9634737cf44e8d75 Sat 2023-12-02 19:03:33 CET Sat 2023-12-02 19:09:40 CET
-2 cf5e03f62a004e578f0bb67b797f59dc Sat 2023-12-02 19:10:12 CET Sat 2023-12-02 19:11:30 CET
-1 5eb5be5949354939a32ac629a807843b Sat 2023-12-02 19:11:52 CET Sat 2023-12-02 19:13:47 CET
0 b04f1e173e414c9d952f710c647002f3 Sat 2023-12-02 19:15:00 CET Sat 2023-12-02 19:17:17 CET
There's a full minute gap where I pressed the power button, waited for it to awake, then hard reset it.
That's great @FalconPDX , thanks for the test! If you're in the mood for testing more (increasing the craziness level heh), how about a new one?
So, for this one, it'd require to use tmux - the terminal multiplexer. In fact, we can split in 2 new tests, let me elaborate below:
[all commands below should be executed as root user]
(A)
(1) For this first test, idea is to login to the Deck via ssh, and run tmux - it should show you a new terminal with a green bar below. So please execute: dmesg -w > /root/dmesg.sleep
(2) Then press the following key combination: ctrl_b and then, the letter d - it should detach from tmux, getting you back to the ssh terminal. Then, please run echo mem > /sys/power/state. Wait for 30 seconds and try to wakeup using the power button - if that doesn't work, force-reboot as you're doing.
(3) Either if the Deck woke-up successfully or not (and required a new reboot), please attach the file /root/dmesg.sleep here so we can take a look.
(B)
(1) This one is quite similar to the first test, but we'll explore the avenue that the Deck is not sleeping, and if so, it can execute commands during the whole time, but GPU and network are dead. Yeah - it's a long shot heh
So, ssh to the Deck and open tmux again.
(2) In the tmux session, run the following: sleep 90 && echo 1 > /proc/sys/kernel/sysrq && echo c > /proc/sysrq-trigger.
After that detach quickly from the tmux session using ctrl_b and then, the letter d.
(3) Finally, back to the ssh terminal, run echo mem > /sys/power/state. Wait for 3 minutes.
(3a) If the Deck reboots, it means a kernel panic was triggered (by your sysrq command above) and a kdumpst log should have been collected. Take a look in journalctl -b | grep kdumpst to see if there was any log saved, and if so, please attach the tarball.
(3b) If after 3 minutes Deck is completely black-screened, try to wakeup and if it fails, just force reboot. In this case, the test didn't work.
Sorry for the bulk of instructions, hope it helps us to figure out if there is any issue at the kernel level. Thanks again!
Definitely down to help out as needed to hopefully find the cause of the issue. Also appreciate the clear guide - not as proficient with embedded systems as I used to be!
A)
I've attached dmesg.sleep below (with .txt extension appended to get around github's file type restrictions)
dmesg.sleep.txt
B) No kernel panic was triggered, and the deck never woke up unfortunately - had to force reboot it in the end.
Let me know if there's anything else I should try!