Flatpak build

[LeandroStanger]

Please, consider supporting Flatpak and publishing OpenTESArena on Flathub.

[afritz1]

Hi. I've never used either of those.

Flathub is primarily intended as a service that is used by app developers to distribute their apps.

I'd prefer someone with more Linux experience do this, but it looks like I need to be the one that makes a PR on the Flathub repo.

Sounds like a good idea, but I think it's low priority given how little there is to do in OpenTESArena at the moment. Do people have to compile the Flatpak build before they use it? I don't understand how else one build would work on all distributions.

[mdmallardi]

Properly compiled flatpak would include all required libraries for the application within them, all the way up from the stdc++ libs to wildmidi.

[ghost]

When this issue gets solved, can we submit the game for inclusion into Athenaeum?

[afritz1]

I don't really want to try distributing builds on other services until the engine is playable and/or close to 1.0. Maybe once I can argue it's a 'game'.

If you look at the GitHub release download stats, there aren't even 80 Linux downloads per release yet. Maybe putting it on more platforms will increase that, but that's beside the point. I just don't think the engine is complete enough to start distributing it on other platforms yet.

[Digital-Monk]

Has Linux been deprecated for the time being? I can't find a 'releases' directory or anything else with a Linux tarball. I was really excited to see the 0.11.0 release announcement, but all of my machines are 64bit Linux-based (Mint, ie sorta-Ubuntu, ie sorta-Debian). Is there a problem building Linux releases? Something I could try to resolve (not high hopes, but if I'm gonna ask, I want to offer)? Been wanting to see Arena remade for, I dunno, 15 years? Really want to keep Linux in the release set.

[afritz1]

@Ragora has been generously volunteering to do the Linux builds so far. I do the Windows builds and I could try doing a 64-bit Ubuntu build but I don't know I would get the compile options all correct.

[afritz1]

I added a Linux build for 0.11.0. I don't have WildMIDI set up for it though. Hope it works :)

[Digital-Monk]

First, thank you for running the build. I know it's non-trivial and not particularly your thing, and I fully understand if you don't want to look into any issues there. But, just in case...

I'm getting an odd crash on Linux64 -- Illegal Instruction. Fired it up in GDB, which I'm not really proficient in, but I was at least able to determine that it's crashing before it ever gets to main(), which means it's in the initialization code for globals (ctors of global objects, usually).

The specific failure line appears to be (compiler generated, so not sure how useful this is gonna be, but...): 0x00005555555a3afa in std::vector<bool, std::allocator<bool> >::vector(std::initializer_list<bool>, std::allocator<bool> const&) [clone .constprop.424] ()

Backtrace gave me this: #0 0x00005555555a3afa in std::vector<bool, std::allocator<bool> >::vector(std::initializer_list<bool>, std::allocator<bool> const&) [clone .constprop.424] () #1 0x00005555555a401b in __static_initialization_and_destruction_0(int, int) [clone .constprop.421] () #2 0x00005555555a3f8d in global constructors keyed to 65535_0_ArenaAnimUtils.cpp.o.293520 () #3 0x00005555556f57dd in __libc_csu_init () #4 0x00007ffff7a05b28 in __libc_start_main (main=0x5555555a3250 <main>, argc=1, argv=0x7fffffffde68, init=0x5555556f5790 <__libc_csu_init>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffde58) at ../csu/libc-start.c:266 #5 0x00005555555a9cfa in _start ()

Maybe the "global constructors keyed to ...ArenaAnimUtils.cpp..." is helpful?

[afritz1]

Hmm. I think ArenaAnimUtils appears the first alphabetically, so it is supposedly the first one to get initialized. In that case it's probably a problem with linking or something like that, not specifically global variables. I don't know how to fix this :(

Is there anything more you can find about the "Illegal Instruction" being used? I might've compiled with -march=native, I'm not sure. I know that SSE4.1 is enabled... maybe that is what's breaking it.

[afritz1]

Okay, so here's what I compiled the Linux64 build with:

SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.1")
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx")
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Ofast -march=native -ffp-contract=fast -flto=full")

I'll assume that your CPU doesn't support AVX or SSE4.1, so I'll turn those and -march=native off and make a new build. I'll get it uploaded in ~15-20 minutes. Let me know if it works.

Since OpenTESArena uses a software renderer, it's important to get better CPU instruction optimization, but it's more important that it runs ;)

Edit: okay, the build is up. I enabled -g and the executable is about 26 MB now but it should work!

[Digital-Monk]

Heh. I am running on an old Xeon, so some of those "advanced" architectures might not work for me. Internet is flaking in and out for me here, but I just got cpuid installed...

SSE and SSE2 are present SSSE3 extensions are present (maybe a typo in cpuid? dunno) SSE4.1 and SSE4.2 extensions are present AVX extensions are present

However: AVX2 is not present AVX512F, AVX512DQ AVX512IFMA AVX512PF AVX512ER AVX512CD AVX512BW AVX512VL AVX512VBMI AVX512_4VNNIW and AVX512_4FMAPS are all missing

AVX/YMM features (0xd/2):
      AVX/YMM save state byte size             = 0x00000100 (256)
      AVX/YMM save state byte offset           = 0x00000240 (576)
      supported in IA32_XSS or XCR0            = XCR0 (user state)
      64-byte alignment in compacted XSAVE     = false

So, I've got a lot of the CPU features, but not all of them... (Haven't kept up with AVX, so not entirely clear on how surprising it may be that AVX2 and those particular commands are missing)

[Digital-Monk]

WOOT!

Working on my end!

(Had a brief glitch with my update scripts...)

Extremely minor point, but the .tar.gz doesn't include the opentesarena-0.11.0-Linux64 path and unpacks directly wherever you are. Doubt it will be a serious problem for anybody though.

Looks awesome! Glad to have water and statics and critters! Thanks!

[afritz1]

Glad to hear it. I was being a derp by not realizing I made the build with architecture-specific settings. I never have a problem with uncompressing .tar.gz on Windows because I just use the 7-zip option to automatically make a new folder and put the contents in there.

Edit: it was bothering me, so I fixed the .tar.gz folder nesting.

[Digital-Monk]

I can do the SSE4.1 and AVX. No idea how common or uncommon those are on older hardware, but my Xeon is hardly new... Not sure what else -march=native may be trying to use. Definitely understand the benefits of auto-parallelization and streaming SMD opcode use for what you're doing.

Um, I also need to say that "Thanks!" is totally inadequate for the amount of joy that I just experienced running willy-nilly through the woods and streams and islands and into the stores and remembering just how effing awesome this game was (even with all the limitations of the time) and thinking about how awesome it could be now with the RAM and CPU constraints essentially removed and a modern movement system and (in the fullness of time) a mod system so stuff can be tweaked up. Seriously, I can't even...

It's possible that this is all caught up in the glow of memory that I played this game with my first girlfriend long, long ago when it was new, but honestly I think it's just the anticipation of seeing this game fulfill the potential that it offered. I think it had a better world generation and map system than any of the later games (maybe specifically because it was simpler geometry, but I don't care -- it WORKED and it worked well).

Anyway, I mean all of that when I say "Thank you!"

[afritz1]

Hurray! \o/

[Digital-Monk]

For anyone feeling adventurous, I strongly recommend setting up the build chain and building it for your local hardware.

For me, on Linux Mint (Ubuntu derivative), that meant:

sudo apt install cmake libopenal-dev libsdl2-dev libwildmidi-dev

to get the dependencies. (I also installed libsdl-dev and wildmidi by accident along the way, but I don't think that's necessary.)

You may also want to sudo apt install cpuid and then use the cpuid command to find out what capabilities your processor has. Output is huge, grep is your friend...

In my case, I was able to add aftritz1's flags back into CMakeLists.txt before running cmake .. (with one slight variation -- cpuid says I can support sse4.2, so I changed it to that).

The resulting build is almost twice as fast. Definitely worth the effort if you have any experience building projects from source.

(I've got WildMIDI compiled in, but probably not configured properly. I don't care about music, so it was low priority to me...)

EDIT: I'm a doofus... Didn't realize that -march=native turns on every conceivable spiffy feature that your local CPU can handle, so once you've turned it on, you don't really have to worry about cpuid and avx and sse4.2 and so on and so forth... And for anyone else like me who was behind the curve on this, this would also explain why -march=native gives best possible performance on the machine it was compiled on, and frequently won't run at all on other people's. Haven't really had to worry about this degree of optimization in a long time. (But now I'm thinking more seriously about those Linux distributions that build everything from source with -march=native... Wouldn't expect the doubling in speed that I saw with TESArena, but even a 10-20% would be pretty slick.)