heads
heads copied to clipboard
linuxboot
keyboard backspace bug for x230-hotp-maximized (x230t)
So I was excited to get heads installed and my nitrokey setup. But then realised I have some sort of bug on all three laptops. No matter the OS, if its linux it locks the screen or just doesn't do anything. Same issue on 1 x230 tablet and 2 x230 notebooks.
How can I go about fixing this? I'm not a seasoned developer so please have mercy on me.
Thank you
@Kawaiisu what is your backspace bug?
I had the following bug on x230t but in OS (QubesOS) not under Heads, if your bug is that the backspace is not linked to the right keymap for some reason: https://github.com/QubesOS/qubes-issues/issues/3306
Please describe actual behavior and expected behavior, which OS, and if behavior is under Heads or not. The operating system being kexec'ed, Heads should not have any influence on the OS behavior. We are interested in the behaviors happening inside of Heads.
Hi,
I went to powershell from Heads and they backspace and backspace keys seem to be fine.
Then I tested debian 10 live and found that the backslash and pipe key "\ |" seems not to function. As well as Backspace.
Again in Qubes OS, same as debian 10, the backspace or backslash doesn't seem to function.
On Tails backslash does nothing but backspace in this instance locks the screen.
I visited this: https://github.com/QubesOS/qubes-issues/issues/3306
Am I supposed to overwrite xmodmap_x230t.txt from Dom 0?
But what about every other operating system? Doesn't it mean there's something I need to do in heads to make sure its permanent across all operating systems? We use many live operating systems.
Thank you.
@Kawaiisu The problem was never resolved. Last time I checked with QubesOS, fix was temporary, and was not able to map it correctly over reboot and was randomly showing back. Looked for help, found none, gave up.
AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all). The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.
If you find the solution elsewhere, please link to the fix. It seems that no one cared, including myself, since I do not use my x230t but to test Heads builds. AFAIK that QubesOS issue included all the pertinent information that I had found from that time. If you find something new, please ressucitate that issue, since as you discovered, is still present. Also, as you saw, the issue is not having any relevance to Heads, since the behavior is not present.
So the issue here is irrelevant.
AFAIK, the issue is relevant to x230t alone, not x230 (I have not that problem at all). The problem should be the same for all x230t being detected as x230 and having issues with keymaps ins OSes who thinks the x230t is a x230. That seems to be the issue, being present across all Linuxes. If there is one operating system not showing the behavior, then that OS fixed the issue and the interest of checking what is different from that OS to others showing bad behavior becomes interesting.
If you find the solution elsewhere, please link to the fix.
@tlaurion Could you use the x230t coreboot config for x230t? Skulls is building separate images for x230 and x230t because the devices are not exactly same. https://github.com/merge/skulls x230t support was merged in coreboot on 03.2020 here: https://review.coreboot.org/c/coreboot/+/38482
It would also make more sense for people who switch from skulls to heads to flash a x230t image when they had before also a x230t skulls-coreboot image.
@ fhvyhjriur Heads is a community project. There is a porting guide on heads-wiki. As you saw for #925 its not really useful neither for myself nor others to do ports for hardware I do not use. The people that have needs should be the one pushing changes. This is mentality of open source and is called Selfish Contributionism. Maintainership is the main problem of open source projects. It would be counter productive that I accept to bring a board to a project I do not use, for whichI will then be asked to maintain. It won't happen.
Consequently, I will not do the port for x230t but will invite x230t, t400 and other not supported board owners to propose pull requests.
And will hold my stance on maintaining and trying to continue to do quality work on my community time, for which i'm not paid for. Hope the message reached its audience.
Free software doesn't mean free as in free beer. Everyone has 24h in a day. If it's important for you, you will take the required time to make it happen, asking questions, proposing code; not requiring others to do the work at your place. This is how it works.
I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.
Letting people take their fish lines instead of providing unlimited fish supply: cp config/coreboot-x230.config build/coreboot-ver/.config cd build/coreboot-ver/ make menuconfig (do config changes needed) make savedefconfig cp .config ../../config/coreboot-x230t.config cd - cp -r boards/x230/ boards/x230t/ mv boards/x230t/x230.config boards/x230t/x230t.config vi boards/x230t/x230t.config (change path for coreboot-x230.config to coreboot-x230t.config) make BOARD=x230t (adjust until it compiles) git checkout -b x230t git add board/* config/* git commit git push create PR.
@tlaurion I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.
You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.
EDIT: When i wrote the text, your last edit of your text did not had this line: 'I recognize, though, that if x230t is detected as a x230 because of coreboot, x230t board definition might be the answer to upstream bug.' Yes, this was the reason why i wrote what i had wrote before.
@tlaurion I also dont have a x230t at the moment. It was just a idea how @Kawaiisu could be helped. The x230t have for example some different gpio compared to the x230. You wrote that you had a x230t for testing images.
You asked for a link for a possible solution. Thats why i linked the x230t coreboot commit. That would fix the problem that the x230t is been detected as a x230 when a x230 coreboot images is been used. I thought you maybe do not know that it was merged in 03.2020 to coreboot and it would be some help to mention this.
I did'nt know. That would then be linked to x230-maximized images, being pushed to coreboot newer version, which are still not merged upstream under Heads. My comment is still relevant. Code should be borrowed, modified, tested functional, and then proposed for merged, tested and then upstreamed by the people who need the changes.
@tlaurion I watched https://fosdem.org/2020/schedule/event/selfish_contributor/ My "selfish motivation" bringing up a idea how to maybe help @Kawaiisu and all other heads users is because heads is a way to do important work as a investigative journalist that travel around the world and have to leave the device for a moment at some place that is not secure. @Kawaiisu is a new user that have register fresh to github to report this problem. So i expect that this user needs heads to do something important to the society and needs help with that. I want to live a life in a knowledgeable society with free access to free(copyleft) knowledge. I am also "selfish motivated" to support old hardware because of freedom(low purchase price, low to none dependencies to startup based on closed source software) and environmental benefits. My "selfish motivation" in for example getting broken printers from electrical waste and soldering out their 8MiB and 16MiB spi chips and putting those SPI chips in thinkpads that have 4MiB or less. To get heads support for such devices is something i am interested in after learning facts about this world. I am not a developer and i am not good at writing code. I depend, like mentioned in the https://fosdem.org/2020/schedule/event/selfish_contributor/ video, mainly on commit messages that explain what the commit have done. And because T400 was mentioned - i have spend happily some hours into taking a T400 apart and writing with a external programmer some heads testing-image and reporting if its working or not. I would do this work again to help adding support for T400 and other devices that are not supported in heads recently. Adding heads-support for all x86_64 devices that coreboot support and that can run in perfect case with 100% free software except cpu-microcode is part of my "selfish motivation". It would be my "selfish motivation" until RISC-V based portable devices are available at the same price and amount like ~10 year old Thinkpads that heads support recently. At least until devices like the https://en.wikipedia.org/wiki/Novena_(computing_platform) and now https://mntre.com/reform get the same produced amount like the thinkpad **00, **10, **20 and **30 series combined together.
At some parts of this world the 11.01.2021 have begun. This is the 8th year since the death of https://en.wikipedia.org/wiki/Aaron_Swartz . My "selfish motivation" is to help the world to be a better place for people with such a free mind like Aaron had.
@fhvyhjriur
Could you use the x230t coreboot config for x230t?
Yes. My Heads test device is a 230T. As far as I can tell is the only difference between the x230 and x230T coreboot configurations is gpio33 changed from output to input. This probably has to do with using the touchscreen as an input during boot up.
@Thrilleratplay did you figured out why backspace was not functionning properly on Qubesos and other OSes?
@Kawaiisu first question first. Is the bad behavior present with stock Lenovo BIOS on live cds?
@fhvyhjriur lookin at coreboot changes, I doubt sincerely that the changes would resolve the keyboard issue.
With that first question here answered, avenues of solutions can then be nrrowed down.
Vivid memories for Swartz. Sympathetic to the cause. Thanks for reminding me why we do this. Just remember time is a limited resource, I don't intend to be rude.
@tlaurion I haven't had an issue with the backspace. I'll try the latest build tomorrow
@tlaurion Open source projects are rife with burnout and you put in a considerable amount of work to Heads. It is important to be reminded why you it. The personal reasons that drive you to contribute so much to Heads are your own. I understand the importance and do what I can with my time, resources and energy but pales in comparison to you do to keep this project going. Thank you.
@Kawaiisu I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries.
What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards.
I still have to test this myself.
@tlaurion @Kawaiisu I may have reproduced this. I am confused on whether it is just the backspace key or all keys that do not work? I flashed the latest x230 maximized build artifact from CircleCI and everything seems to work fine but occasionally after a reboot, the keyboard does not seem to be initialized. It also only seems to happen when rebooting from the emergency shell either with the command reboot or ctrl-alt-del.
EDIT: Oops, I used a test build. Let me try one from master.....and osresearch does not allow/have any projects in circleCI because there is only a 404 when I click on projects.
EDIT 2: Updated master on my fork and used the CircleCI build heads-x230-maximized-v0.2.0-1006-g6bc40d7.rom. If I boot into a USB image of Debian 9, I can go into a prompt and still use the backspace and pipe characters on the keyboard.
@tlaurion @Thrilleratplay
i remember this bug with backspace key sorry but its not only Qubes same issue with x220/x220t (flashed year ago) it works in heads shell it lock screen on gnome desktop (lastest ubuntu) it doesn't work everywhere in terminal/chat/text editor never seen this bug on clean coreboot or stock bios. replaced keyboard - nothing changed maybe really xev or xmonad can help I use classical six row kbd
I would never give more than $ 50 for these laptops, ebay disappointed me a lot. out of five, I have only one alive (librebooted x200 from Vikings)
RIP Aaron :(
Thank you all so much for your generosity.
Sorry for my delay. I hope you're all finding ways to prepare and maintain your safety during these trying times.
@tlaurion : you asked "Is the bad behavior present with stock Lenovo BIOS on live cds?"
I will double check this on a stock device and post my results today.
Also I think you're correct that there isn't this problem on the notebook non-tablet version of the x230. I'll double check this as well.
@Thrilleratplay you said "I am going to assume you did not do any keyboard swap or Thinkpad-EC style EC modifications to allow third party batteries."
No mods. I did do a backlit keyboard on one, but went back to stock keyboard.
@Thrilleratplay: "What country are all of these laptops from? If they are not all North American Models, I wonder if something like the default grub keymap does not work on certain regional keyboards."
Laptops were purchased from a recycler. But there was spanish post-it notes from the previous owners. Its possible they came from South America, would this matter? What could I do to figure this out? I think the recycler would not tell me for competitive reasons.
I will go through in detail everything else mentioned above in case I missed anything.
@Kawaiisu I ask about where they are from because there are regions variants of keyboards (Japanese, Russian etc). Normally these are obvious, other times it could be subtle like the currency character not being $ if you are in the US. A few keys could be mapped differently on these variants for backspace and pipe. Although, I think the backspace would be the same on all of these.
@Thrilleratplay @tlaurion
Okay I've tried both tails and debian 10 live on both stock x230 notebook non-tablet version and x230 tablet versions. Backspace and and backslash work normally. I also swapped out the tablet with a backlit keyboard and backspace and backslash still work as expected.
Also not sure if it will become relevant but want to note it so I don't forget: Another X230 notebook I flashed using make BOARD=x230 and x230-flash rom creation process. On that system backspace and backslash work just fine.
The system that's giving me the problem is an x230 tablet and I used make BOARD=x230-hotp-mazimized. Not sure if the rom difference is relevant.
I am going to now flash the stock x230 tablet and notepad with the x230-hotp-mazimized rom and will report if the backspace-backslash problem comes back.
@tlaurion about whether I have country specific keyboards, they seem to be all US, except 1 backlit keyboard on the problem computer does have a random extra euro symbol on the 5 and % key. So three symbols on this one key. Which is strange. The other backlits don't have this and seem to be US-only. Could be something here.
Okay I'll get to flashing and report back my findings.
But first I have a question about preparing the x230-hotp-maximized roms.
In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:
"Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop."
I know I'm doing something really noob here. A hyphen or space or something is missing or needs to deleted, please have mercy on me ^_^.
To produce the roms while getting around the error we skipped this part and manually ran wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin then made the roms after.
But if I know why the error was happening this will speed up our process.
Thank you. I'll get to flashing now.
@0rb677 @tlaurion @Thrilleratplay @fhvyhjriur
Greetings everyone. I'm reporting back my findings.
I newly flashed two laptops, a stock x230 Tablet and a stock x230 Notebook non-tablet.
The notebook's backspace and backslash work fine. However on the Tablet backspace and backslash don't work as intended. Locking the screen in Tails for instance.
I did test both stock laptops prior, they were both working as intended on multiple OS before the flash.
So at this point I'm stumped, is this a heads problem or something else?
Not really sure how to proceed here. Any advice would be greatly appreciated.
Thank you all for your time and insight. Be safe.
@Kawaiisu Just to clarify, all of your x230 work as expected but the x230T still has an issue with the backspace key? What OSes have you tried after flashing Heads?
Lets try to figure out what the OS sees (link to helpful answer). At the moment I am on a x230 that is running coreboot and Arch Linux.
If I execute sudo showkey and press backspace. The key code is:
keycode 14 press
keycode 14 release
sudo showkey -s and press backspace. The scancodes is:
0x0e
0x8e
If the same OS and version of Heads is installed on both your x230 and x230T, it would be more beneficial if you compare the the results between those machines. Hopefully this will narrow down the issue.
Hi @Thrilleratplay
Yes you're correct on the x230 backspace and backslash works fine with Qubes.
But x230T with Qubes backspace and backslash doesn't work.
On x230, running sudo showkey and pressing backspace shows:
keycode 14 press
keycode 14 release
Then running sudo showkey -s and pressing backspace shows:
0x0e 0x8
Then running sudo showkey and pressing backslash shows:
keycode 43 press
keycode 43 release
Then running sudo showkey -s and pressing backslash shows:
0x2b 0xab
On x230T, running sudo showkey and pressing backspace shows:
keycode 152 press
keycode 152 release
Then running sudo showkey -s and pressing backspace shows:
0xe0 0x12 0xe0 0x92
Then running sudo showkey and pressing backslash shows:
keycode 139 press
keycode 139 release
Then running sudo showkey -s and pressing backslash shows:
0xe0 0x1e 0xe0 0x9e
Does that help?
@Kawaiisu It sounds like a bug in Qubes and now that you know what the keys are, remapping seems like the easiest fix. I am not familiar enough with it to suggest an appropriate work around. @tlaurion Can you suggest a way to globally remap keys in Qubes?
@Thrilleratplay @tlaurion Want to give update. The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.
Also on the 230T, in heads powershell backspace works correctly. So I'm confused, why do all OSs work on a stock X230T, but when I flash heads, suddenly no OSs work?
Are we sure this is a Qubes specific problem? Isn't Qubes natively Xen? Should I test more OSs?
@Kawaiisu i'm confused on your conclusions. You say that QubesOS, Tails and Debian 10 live OSes: the behavior is the same. I guess the behavior is bad? Or you mean by flashing Heads, the behavior changes to bad?
@Kawaiisu : can you send me a rom backup of lenovo x230t? I cannot guarantee when I will invest time in this, but my first debfugging steps, as said prior, would be to test behavior on stock BIOS. My hypothesis is that OSes are detecting keyboard and set mappings based on detected x230. Which doesn't match x230t. Another note: QubesOS is based on Xen, yes, but the underlying dom0 OS for QubesOS 4.0 is fedora-25. So issues disovered here are fedora-25 related for QubesOS. Or related to Xorg keyboard, or underlying mapping to keyboard config.
@Thrilleratplay I left the ticket as unfixed under QubesOS with all steps I took in the past trying to fix the issue. As of now, i'm still into uncertainty on what is the problem to replicate. On my past experience on X230t with QubesOS, the behavior was random. It was functional at first install (I could change LUKS key passphrase doing backstpace) and then after OS install, I couldn't. The issue I opened (QubesOS 3.2) was closed because noone else was confirming the issue. So the user base of X230t seems quite small. Noone else replicated the issue; noone +1. I investigated as far as I could. And left the x230t alone.
The heads related ticket is : https://github.com/osresearch/heads/issues/827#issue-692472819 The QubesOS related issue is: https://github.com/QubesOS/qubes-issues/issues/3306
I think the problem is generalizing to all OSes. And the fix to test is this: https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-359933561 Unfortunately, it seems that it was not persistent(the problem came back on next boot? I can't recall. 2018, after all!), since I after that posted https://github.com/QubesOS/qubes-issues/issues/3306#issuecomment-366265534 ....And then, lost interest in the X230t, which I use to test flashes, and where Heads backspace works, where there is no added layers of overcomplications or guessing and overides of keymaps.
So I will ask again:
- Is the behavior present on stock Lenovo bios for X230t?
- Is the behavior limited to Heads?
If the behavior is limited to Heads, which is a duplicate of the X230 coreboot config, then maybe coreboot exposing the board as a x230 in dmi tables might be the culprit. Maybe the remapping of keys in OSes are basing themselves on exposed X230 mappings instead of X230t.
Please give feedback on 1 or 2 above. Instead of trying to resolve the issue and waste time. Troubleshooting steps from simple to complex is key, else you will loose yourselves. If we know the behavior is different between installed OSes from stock BIOS vs Heads, then we might be able to get the differences in assumptions and blame the x230 config inside of the x230t board. And start from there.
If I missed your conclusions, please re clarify and sorry to have missed them; I have not see clear statement of differentiation between 1 or 2 being the cultprit.
@Kawaiisu I asked what OSes you had tried, as the respond was only pertained to Qubes, I assumed the issue was isolated to Qubes.
I booted Debian 9 on my x230T with maximized Heads and do not have this issue. The shell I entered from the Debian Graphical install does not have showkey. The best thing I can find is to run more /proc/bus/input/devices. On my x230T, the keyboard name is "AT Raw Set 2 keyboard". Interestingly, my x230 is "AT Translated Set 2 keyboard". Looking at them, they look like they are interchangeable.
Steps to take:
- Boot into Debian using the USB
- Go to a shell
- run
more /proc/bus/input/devices, scroll until you find your keyboard. - Google search the name and maybe the version ID to see if there are any issues with that specific keyboard.
- Afterwords, for shits and giggles, swap the keyboards on a working x230 and the x230T and see if that fixes it. If this works and no one knows what to correctly map the bad keyboard, it may be easier to buy a replacement keyboard at this time.
@tlaurion I saw the solution you posted in Qubes issues #3306 using xmodmap but, unless I am mistaken, this would only work in X11, not in a terminal if x11 failed to start or the user enters another TTY session by pressing Ctrl-alt-F2 (this may not be an issue in Qubes).
@tlaurion apologies, let me correct my poor grammar:
The bad behaviour is exactly the same for the X230T on Tails and Debian 10 Live OSs. It gives the same key codes and scancodes as Qubes on SSD as cited above.
You asked:
- Is the behavior present on stock Lenovo bios for X230t?
The bad behaviour isn't present on a stock lenovo bios for the x230t. I test qubes, debian live and tails.
- Is the behavior limited to Heads?
Yes the bad behaviour of incorrectly mapped backspace and backslash only appears only after flashing.
I tried to upload .zip directly to github, either its not supported or my browser doesn't allow. I was able to upload here:
https://ufile.io/3vrd09o2
Inside is both heads that I flashed to the x230t and the factory bios before should you need.
@Kawaiisu Could you give the skulls x230t image a try and report if you have the same issues? https://github.com/merge/skulls/releases/tag/x230t-0.0.2 Probably the best way to do the test would be to first install the x230t oem backup you have made at some point, test if the issue is gone with the OEM again (never know if EC does the problems) and then follow the skulls installation howto. Would be interesting to know if the issue is gone with that skulls-x230t-image that is made for x230t. If the issue is gone with the x230t skulls image, you can flash the skulls-x230 https://github.com/merge/skulls/releases/tag/x230-0.1.11 release to the x230t to find out if it have to do something with that and the issue appear again.
@Thrilleratplay
Upon your suggestions I ran on debian 10:
more /proc/bus/input/devices
It showed on the x230t:
AT RAW set 2 keyboard" and ID: "ab84
I googled around and nothing came up relevant to this issue.
I swapped out the keyboard from a x230 notebook where the backspace is working on to the x230t, and the backspace still doesn't work. I ran more /proc/bus/input/devices and the readout is identical to before AT RAW set 2 keyboard" and ID: "ab84
You said heads maximised is working correctly with your x230t, do you think it's possible I built heads incorrectly?
Beforehand when I built heads, I was using make BOARD=x230 and x230-flash, I had to copy the original OEM roms to the flashing device and unlock them or remake them into new roms.
But there was a key difference when doing BOARD=x230-hotp-maximized, where I never unlocked the OEM roms from the original x230t. Instead the roms were downloaded. Is it possible I downloaded the wrong roms?
In the x230-hotp-maximized.config there's a part that says "call - blobs/xx30/download_clean_me.sh." which is commented out. So I removed the comment # and running make Board=x230-hotp-maximized gives us:
Wrong gawk detected: 4.2.1 /home/user/heads/boards/x230-hotp-maximized/x230-hotp-maximized.config:65: *** missing separator. Stop.
I couldn't figure out what the custom invoke or call command was here so I tried to get around this step by manually running
wget https://download.lenovo.com/pccbbs/mobiles/g1rg24ww.exe && innoextract g1rg24ww.exe && python ~/me_cleaner/me_cleaner.py -r -t -O ~/heads/blobs/xx30/me.bin app/ME8_5M_Production.bin
then made the roms after.
Is it possible I missed a key step involving a difference in the way to make the roms for the x230 versus the x230t?