Linux Support

[Utumno]

Bash should be able to run on linux. Opening this to start investigating.

• [ ] Case sensitive filesystem - oopsie woopsie, Bash has absolutely no notion of that

• [ ] Proton awareness - so that the next bullet point is obsolete for many (most?) users

• [ ] A (better) way to configure paths, e.g. of the documents folder - ref #572, we do not want to continue adding to bash.ini (because of #250)

• [x] app buttons code is tightly coupled with ~~_winreg~~ win32api - Linux does not really use app buttons code anymore, but that just points to us needing to rethink these entirely (#570)

• [ ] hunt down all hardcoded backslashes (duh) - ref 2e65a933e1a53b582a7d3c30600171f7ad62ad6c

• [x] move winreg from bass and to an environment detection module - cross platform

• [x] WindowsError, duh:http://stackoverflow.com/q/1210118/281545, http://stackoverflow.com/a/8890190/281545, http://stackoverflow.com/a/2678797/281545, http://stackoverflow.com/a/22027543/281545 -> edit done see #252

• [x] this (traceback solved in a9751915e101bf0ba6261e66dce1ff5d640f3e5d but check also warns) :

  $ python2 /mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/Wrye\ Bash\ Launcher.pyw -o /mnt/win/GAMES/TESIV/Oblivion
  No translation file for language: en
  01:33:47 AM: Warning: Mismatch between the program and library build versions detected.
  The library used 3.0 (wchar_t,compiler with C++ ABI 1008,wx containers,compatible with 2.8),
  and wxPython used 3.0 (wchar_t,compiler with C++ ABI 1002,wx containers,compatible with 2.8).
  Traceback (most recent call last):
  File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/Wrye Bash Launcher.pyw", line 88, in <module>
    bash.main()
  File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/bash.py", line 363, in main
    import bosh
  File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/bosh.py", line 55, in <module>
    import balt
  File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/balt.py", line 666, in <module>
    import windows
  File "/mnt/win/Users/MrD/Desktop/ZZZ/wrye-bash/Mopy/bash/windows.py", line 31, in <module>
    from ctypes.wintypes import MAX_PATH
  File "/usr/lib/python2.7/ctypes/wintypes.py", line 23, in <module>
    class VARIANT_BOOL(_SimpleCData):
  ValueError: _type_ 'v' not supported
  

~~Of course the biggest hurdle is the dlls... If it proves easy we can compile them to .sos so much the better~~ All DLLs dropped by rewriting relevant code in Python:

• [x] libloadorder - #295
• [x] libbsa - #339
• [x] libloot-python - #431
• [x] CBash - #530

Blockers:

• [ ] #250 - if we want to deploy to a single installation point in /usr or /opt, we need profiles
• [x] #460 - most distros don't package the libs we need for py2 anymore
• [ ] #572 - weird setups and paths are even more common on Linux, we need to solve this once and for all (and in a profiles-friendly way, no adding to bash.ini!)

[nycz]

i've been thinking about the problem with case (in)sensitive paths a bit since that is the by far biggest hurdle to get bash running on linux. the only decent option i can come up with is generating and keeping a dict of lower-case-to-mixed-case path pairs that bash's internal path system deals with.

basically this:

1. something wants to access a file in Directory1/dir2/DIR3/File_Name
2. change the whole path to lower case
3. list all items in the root path on disk, change them to lower case and find directory1
4. add the unchanged case version of directory1 (let's say DIRectory1) to the output path
5. go to DIRectory1 and repeat until you have the full path

i have no idea if this is as slow as it sounds, but it should be fairly easily cached. also, i'm thinking this should only be done this way on case-sensitive file systems so hopefully windows shouldn't be affected. one fairly significant problem would of course be case-sensitive file conflicts, for example if someone manually adds the directory Actors where actors already exists, but i don't see an easy way around that. my best bet would be to add a check for duplicate files/directories or something.

[Utumno]

if someone manually adds the directory Actors where actors already exists, but i don't see an easy way around that. my best bet would be to add a check for duplicate files/directories or something.

And that's where I stopped reasoning about that

What I think is a better approach is to refactor "bash's internal path system" till it's absolutely clear where the fine line between the path handling and the system calls limit lies - pathlib has separate mixins for abstract paths and concrete paths and Pitrou certainly though deep about that ;)

So help needed in refactoring - I filled up an issue ( #368 ) and my current wip branch contains a new data structure that is very related to the problem (LowerDict) - a very tricky merge I have not found the time to push to dev. Keep in mind that it took me couple of years to finally realize what's the right way to address that (simpler) problem - there are many details that I can't really convey but LowerDict is the tool for that job.

Keep also in mind that refactoring in Path class led to a dramatic decrease in memory: https://bethesda.net/community/post/371559

So this is not an easy problem. Bottom line we should revisit this once we've switched to pathlib - and that's way, way ahead. Meanwhile there's a lot to be done.

[nycz]

first of all i hope i didn't come across as arrogant or sounding like i had the silver bullet solution or anything D: i know things are a lot more complicated than they might seem for someone not intimately familiar with the source code and i don't expect this to be solved easily or quickly. this is, however, more or less the one thing that keeps bash from working on linux so the sooner it's done (with or without my help) the sooner i can use (and by extension develop) bash properly on my own computer so i'm eager to do what i can to help, specifically with this.

with regards to your branch and LowerDict, is that so tightly tied to future Path changes that it's not meaningful for me to start diving deep into Path system and pathlib yet?

[Utumno]

first of all i hope i didn't come across as arrogant or sounding like i had the silver bullet solution or anything D:

Haha, not at all, same here

this is, however, more or less the one thing that keeps bash from working on linux so the sooner it's done (with or without my help) the sooner i can use (and by extension develop) bash properly on my own computer so i'm eager to do what i can to help, specifically with this.

Preferably with your help - dunno of anyone else that actually uses Bash on pure linux :P

with regards to your branch and LowerDict, is that so tightly tied to future Path changes that it's not meaningful for me to start diving deep into Path system and pathlib yet?

It's certainly meaningful. I really don't know how to guide you though - the hypothesis that paths are case insensitive is implicit in the code. I think you should start with the hypothesis that case insensitive paths are indeed unique (that is ignore cases like Data/Textures and Data/textures) and investigate if all works - tests...

Bash's APIs are case aware however (both Path and LowerDict) - that's a good thing

I will merge Asap but meanwhile, having a look in there, helping (once merged) to clearly name attributes that use case insensitive data structures (prepend ci_) then hunting down Path (start with cs property uses) is a possible course of action.

But also probably tackling other better defined refactoring issues will make you more familiar with the code (and help with current dev) - for instance here's one I would like someone to take off my plate: #372

[BeermotorWB]

This may a bit exotic, and may be outside of the scope of Wrye Bash as an application, but there has been some work done on an overlay FUSE-mountable filesystem called ciopfs.

The TL;DR of it is it will take a directory (e.g. "/usr/local/Windows Games/skyrim/data/") and overlay it as a case-insensitive union/overlay mount at some other path (e.g. /home/soandso/skyrim/data/), all in userland so it doesn't require escalation of privileges to mount it.

The good news about this is there is (or was) already a ciopfs package for Ubuntu in the main repository. I'm not sure about its availability on other distros, specifically Fedora which is the only distro with any kind of Wayland movement at the time of this writing.

I'm not saying this is the panacea for this particular issue, but it may feed some ideas.

[Utumno]

Good to know @BeermotorWB

[Ember2528]

Recently case insensitive directory support was added to the Linux kernel for ext4 filesystems. It's not a complete solution since people use all kinds of filesystems, but can help with part of that problem

[Utumno]

@Ember2528 all good but we have a more pressing matter at hand - namely wx3 and python3 - hands needed

[Utumno]

Got a new answer on MAX_PATH queries that seems worth investigating https://stackoverflow.com/a/62796371/281545

[monyarm]

Tried running Wrye Bash on linux, and found a few things that need to be changed in order for it to even launch pull all of the windows specific classes in env.py under a

if _os.name == "posix":
    pass
else:

change the ctypes import to :

from ctypes import byref, c_wchar_p, c_void_p, POINTER
try:
    from ctypes import Structure, windll, wintypes
except (ImportError,ValueError): # we're on linux
    Structure = None
    windll = None
    wintypes = None

As importing from ctypes throws an ValueError under linux

[llde]

I think WB on linux is supposed to work only under wine (and it works well, mostly)

[Infernio]

This is the issue for eventually getting WB to natively run on Linux, without wine. The current status of this issue is that we're keeping it in the back of our heads and trying hard not to introduce any assumptions that only hold on Windows, but WB isn't even close to running properly on Linux yet and it's not a focus either. Our current focus is #460, which will help this issue as well with so many distros now dropping py2.

As @monyarm noticed, env.py is currently Windows-specific. It's not supposed to be and eventually we want to refactor it into a package that provides two implementations, one for Linux and one for Windows.

[Utumno]

Thanks @monyarm - @llde no linux support is a goal, albeit an experimental one, as a case sensitive filesystem makes no sense for a game developed on windows. But if the user knows what they are doing Bash should eventually run in linux

EDIT: ninj'ed :P

@Infernio I had a branch somewhere making actually env import from _windows that's the goal (and it will be the _windows/_linux.py that will blow with import error but yeah, 480, 312, 460 ...)

[Infernio]

Was scratching my head why code like this broke on Linux:

dict_of_paths = {}
# dict construction etc. here
some_val = dict_of_paths[some_path] # blows on Linux

Then I remembered:

[infernio@inf-desktop ~]$ python3
Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.normcase('/home/infernio/.local/share/Steam/steamapps/common/Oblivion/Data/Oblivion.esm')
'/home/infernio/.local/share/Steam/steamapps/common/Oblivion/Data/Oblivion.esm'

So Path.__hash__ doesn't work because currently Path._s == Path._cs on Linux, always :P

[Infernio]

We can't just use lower() either - normcase does more on Windows: https://docs.python.org/3/library/os.path.html#os.path.normcase

Edit: we can't even put it into env and change the behavior by OS - at least the usage in bsa_files really does need the other part of normcase's behavior. Edit 2: see 8b6aecb91b8b45003c228635c1f601a2e26a4c57

[ghost]

Hi, I wrote a small patch which adds better support for the XDG Base Directory Specification on Linux.

Also, I have some more (WIP) patches in a local repository based on the 460-py3 branch (managed to install some basic packages and list their contents), but I was wondering whether I should wait until the Python 3 transition is complete before working on them (it seems most major modifications have been made already). Note that I don't have a Windows machine, so I can't really help with the Python 3 transition (no way for me to confirm that changes won't break on Windows).

[Utumno]

Thanks @dsemy :) - not sure I can review that (not sure what XDG_DATA_HOME is) but from a coding style perspective do not use data as a local variable. See https://github.com/wrye-bash/wrye-bash/wiki and especially https://github.com/wrye-bash/wrye-bash/wiki/%5Bdev%5D-Coding-Style - see also the 306/307 releases commit messages for a thorough overview of development (including internal refactoring goals).

Also, I have some more (WIP) patches in a local repository based on the 460-py3 branch

You can push those on your fork so that me or Inf have a look - but yes immediate goal is #460 - there are some bugs listed in that issue that you may be able to debug even on linux

[ghost]

Utumno writes: but from a coding style perspective do not use data as a local variable.

I've force-pushed an amended commit (to keep the log tidy) to my branch which changes the name of that variable to 'app_data'.

Utumno writes: You can push those on your fork so that me or Inf have a look -

I'll keep working on them then.

Utumno writes: but yes immediate goal is #460 - there are some bugs listed in that issue that you may be able to debug even on linux

Sure, I'll have a look at what needs to be done and work on it if I have some time.

Edit: fixed formatting (GitHub doesn't handle inline quoting through mail as it turns out).

[Infernio]

The patch looks fine to me. Not sure if the games (via wine/proton) themselves obey XDG specifications, but at the moment you need to specify paths via CLI anyways, so it's not a big deal either way.

[Infernio]

So, I've pretty much ditched Windows entirely now after my new hardware upgrade and reinstall caused its Start menu to break, leaving me unable to open programs or even shut the computer down without resorting to ridiculous workarounds (Ctrl+Alt+Del > Task Manager > Run Program > wt.exe/explorer.exe/etc.). I still have a small VM for building and testing WB, but beyond that I've switched over entirely. Which means that this issue will become a lot more important to me in 312 :sweat_smile:

[Utumno]

This explains the silence, I thought exams :P - which distro are you on?

I have been busy as you see finalizing 312 stuff that I started tackling at the end of 647 - still trying to figure out the _AParser

[Infernio]

Arch (btw), have been for years.

[Metallicow]

This may or may not point you to the correct answer, but in my experience their is a difference between commanding linux. basically it boils down to this...

debian vs redhat package manager

now my experience has been that the "blue" package manager which is debian as a shit ton of options, is easy to sort, and so on. BUT a "red" manager is fairly complex (they are probably more dealing with security at the root). My choice of flavors happened to be with mageia back in the XP days(they have a "red" base installer) and while debian is great, I choose to deal with the data(or so called other shit) in in simple terms with BAIN style installation.

I'm not sure if this will work all out for everyone in the end but from my experiences it sort of works.

I would like a safe OS installer(red), have a program installer (blue), and top that all off with BAIN installer (grey).

[Infernio]

I'm gonna mark this as done for 312. Linux support is close to 100% usable now, only some QOL stuff and the tool launchers left (which are WIP anyways, what with #570).

[Infernio]

Still need to mark it as supported by removing the need for --unix, will close via a commit that does that.