Titanfall 2 fails to start (Unimplemented Opcode)

[r3muxd]

6443|0x180008561: Unimplemented Opcode (C6) F0 0F C0 01 0F B6 D8 48 8B C6 8B CB 48 D3 E0

on boot in tier0.dll

tier0.zip

[pg9182]

Something which might be relevant for fixing this: https://patchwork.kernel.org/project/linux-arm-kernel/patch/[email protected]/

[pg9182]

Also, note that this was with the Northstar dedicated server + wine64 + d3d11 and gfsdk stubs in the game dir (with d3d11 set to native in winecfg).

[pg9182]

Thanks @ptitSeb, we'll try running it again soon (I didn't notice your commit earlier).

[pg9182]

OK, another one: 1444394|0x188a17fd: Unimplemented Opcode (3B) CA 76 04 49 8B C3 C3 72 57 4D 3B 43 40 75 51 (retf).

[pg9182]

@ptitSeb?

[ptitSeb]

the opcode is a RET FAR. it maybe linked to a 64bits <-> 32bits call / ret, and need some more refactor. I'll probably start working on next dev. cycle (so in 0.1.9)

[Kuratius]

Current release is 0.2.0, what's the current status on it?

[ptitSeb]

Current status of what?

I don't have Titanfall 2 in my collection so I cannot tes myself. I haven't advanced on 64<->32 call as I have no program using that currently.

[Kuratius]

If you tell me your email or a steam account I can buy Titanfall 2 for you as a steam gift, if you're still interested in working on this. It's currently on sale, so it's a good time to do so.

[Happyllama25]

If you just want to test the server, it requires few of the actual game files, there’s a script somewhere that downloads the correct files. You wouldn’t be able to open the game and join the server for full testing though, caveat there.

[ptitSeb]

My steam account is _ptitSeb_. Is this game big?

[Happyllama25]

~70gb off the top of my head

[Kuratius]

My steam account is _ptitSeb_. Is this game big?

I've sent you a friend request on steam. For some reason I can't send a gift using just a name or an email, at least according to the web interface.

This is the project where this was tested: https://github.com/pg9182/northstar-dedicated

According to the readme it's possible to trim the game to 2GB-4GB if necessary, but the base game is fairly big. The idea is to make it possible to run a headless server on ARM.

[ptitSeb]

I'll probably need to remove Doom from my ARM dev. machine then, as it's about the same size. Or Dota2, it's probably big too. Both are working fine now, so it's not a problem to switch to something else.

[pg9182]

• For the dedicated server only (which is the important part -- the goal is to be able to run servers on the Oracle Free Tier, and on Raspberry Pis), you can use https://gist.github.com/pg9182/9a962adbfc27e93237cd14e4523c9da8 to download the 2.5GB of files I've optimized. You'll need to download Northstar and extract it over the downloaded files.
• You will need 3 cores, or it will likely hang during startup due to limitations in the game (1: the game checks for 3 cores in a few places, and while we've managed to get it to start on 2 with a lot of coaxing, it's not reliable and we haven't included the patch in Northstar, 2: there's some screwy threading stuff around rpak loading which we haven't figured out which makes it deadlock if you force it to start on 1 core).
• I recommend Wine 7.0, but anything 7.0+ should work (however, note that ~7.8+ (possibly as early as 7.3) has performance issues with the server, although that doesn't matter if you're just testing).
• My custom wine build in the Docker image is not mandatory. You should disable ShowCrashDialog in the registry. winedbg will probably not work.
• My nswrap wrapper can be compiled standalone, and works on Linux 5.4+. I highly recommend using it, as it'll manage Xvfb and clean up the Wine output. You don't need Xvfb f you have a real X server. To enable Xvfb, use DISPLAY=xvfb (nswrap will handle the env var).
• To test it, you can use a command like ./nswrap /path/to/northstar/files -dedicated with wine64 (and you don't need WOW64 support in wine btw) and Xvfb in your PATH.
• If Northstar v0.11.0 gets released, I still recommend staying on v0.10.x since the crash handler changes may make it more difficult to test.
• The R2Northstar directory needs write permissions.
• Not all logged errors are actually a problem; basically just look for a "registered to master server" and a "mapspawn" log with some ASCII art and a bunch of warnings/errors about AINs without a crash to know it's successful.
• You don't need a real or emulated GPU; my D3D11 stubs automatically loaded in Northstar 1.6+ will do the job.
• Don't hesitate to contact me here or on Discord if you have trouble getting it set up.
• If it's easier for @ptitSeb, they can use the full Titanfall 2 build on Steam and simply test the vanilla client, but I wouldn't recommend it since it's actually more complicated (due to Origin/EA issues), and we are mostly just aiming to run the Northstar dedicated server (which also happens to patch out a lot of code paths).

[ptitSeb]

Ok, I'll start grabbing the optimized server (my dev. arm machine has 8 cores, and I have a wine 7.5 & wine 7.22, both 64bits, ready to use)

[Kuratius]

@ptitSeb I sent you a steam gift with Titanfall 2, I hope it helps.

[pg9182]

also cc @GeckoEidechse

[ptitSeb]

Thank you @Kuratius You I try the optimized server for now or switch to the full version directly?

[Kuratius]

Thank you @Kuratius You I try the optimized server for now or switch to the full version directly?

I'd say try out the optimized server, as getting that working is probably more useful.

But @pg9182 probably has a better grasp on what to do.

[pg9182]

And if you want to connect to the server once it runs, you'll need to forward tcp/8081 and udp/37015 (you can change these, though), and you'll find it in the Northstar server list (see the wiki for client setup instructions for Steam).

I'd say try out the optimized server, as getting that working is probably more useful.

Yes, and even if you do plan to get the full client working, it's a bit easier to test the server.

Oh, and if you do try the client and get an Invalid Name error, that's a known issue caused by changes on EA's side, but send me your Origin UID or a request ID from the logs, and I can manually do a workaround for you.

But @pg9182 probably has a better grasp on what to do.

Yes; I did a lot of the stuff for getting the server working on Linux (docker image, nswrap, file optimization, d3d stubs)

[ptitSeb]

Ok, So I downloaded the server. aunching wine64 NorthstarLauncher.exe -dedicated start the thing, and end with a crash for read 0x0000000000000001 Is that what you have? I haven't tried nswrap yet as I'm trying to see how/where it crash with the minium stuff loaded in memory, to ease the debugging for now.

[pg9182]

with the minium stuff loaded in memory

nswrap will make debugging easier; it helps normalize stuff.

end with a crash for read 0x0000000000000001

I'd need to see the logs around it; I haven't tried running it on ARM since Feb.

P.S. I might take a little longer to respond (if I respond) for the next two hours.

[ptitSeb]

Using emulated /home/seb/wine/lib/wine/x86_64-unix/crypt32.so
[2022-12-11 17:50:40.250] [info] Profile was not found in command line arguments. Using default: R2Northstar
[17:50:40] [info] Enabling hook _GetCommandLineA
[17:50:40] [info] Enabling hook _LoadLibraryExA
[17:50:40] [info] Enabling hook _LoadLibraryA
[17:50:40] [info] Enabling hook _LoadLibraryExW
[17:50:40] [info] Enabling hook _LoadLibraryW
[17:50:40] [info] Command line: "Z:\home\seb\Games\x86_64\Titanfall2_server\NorthstarLauncher.exe" -dedicated +setplaylist private_match
[17:50:40] [info] NorthstarLauncher version: 1.10.9.0
[17:50:40] [info] Loading resource from library
[17:50:40] [info] Succesfully loaded R2Northstar/plugins\DiscordRPC.dll
[*] Loading l0108:fixme:ver:GetCurrentPackageId (0000000019D0FE10 0000000000000000): stub
auncher.dll
[*] Launching the game...
Failed to instantiate discord core! (err 4)
[17:50:41] [info] Enabling hook ReadFileFromVPK
[17:50:41] [info] Enabling hook CBaseFileSystem__OpenEx
[17:50:41] [info] Enabling hook AddSearchPathHookterialSystem
[17:50:41] [info] Enabling hook ReadFromCacheHook11.dll
[17:50:41] [info] Enabling hook MountVPKHook
[17:51:54] [error] Northstar has crashed! a minidump has been written and exception info is available below:
[17:51:54] [error] Cause: Access Violation
Attempted to read from: 0x0000000000000000
[17:51:54] [error] At: filesystem_stdio.dll + 0xe890a
[17:51:54] [error]     Northstar.dll + 0x52df4 (0x179b2df4)
[17:51:54] [error]     ntdll.dll + 0x27fd6 (0x170027fd6)
[17:51:54] [error]     ntdll.dll + 0x60a25 (0x170060a25)
[17:51:54] [error]     ntdll.dll + 0x5e5ae (0x17005e5ae)

It was just a quick test. I'll build nswrap and try it properly...

[pg9182]

Ugh, haven't seen that one before... I'll look into it later today if you can't get it to work. Might also be a good idea to try running it unemulated so you can compare the output. @BobTheBob9 might also be able to help.

[ptitSeb]

Here with nswrap (just the end)

[18:15:37] [info] Registering ConCommand reload_mods
[18:15:37] [info] CreateInterface ENGINE VCvarQuery001
[18:15:38] [info] Enabling hook D3D11CreateDevice
[18:15:38] [info] CreateInterface ENGINE VAvi001
[18:15:38] [info] CreateInterface ENGINE VBik001
[18:15:38] [info] CreateInterface ENGINE VENGINE_LAUNCHER_API_VERSION004
[18:15:38] [info] CreateInterface ENGINE VDataCache003
[18:15:38] [info] CreateInterface ENGINE VPrecacheSystem001
d3d11: D3D11CreateDevice: initializing d3d11 stub for northstar (github.com/R2Northstar/NorthstarStubs)
Using emulated /home/seb/wine/lib/wine/x86_64-unix/opengl32.so
0154:fixme:nvapi:unimplemented_stub function 0x7f9b368 is unimplemented!
0104:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION
[18:15:38] [info] MountVPK vpk/client_frontend.bsp
[18:15:38] [error] Northstar has crashed! a minidump has been written and exception info is available below:
[18:15:38] [error] Cause: Access Violation
Attempted to read from: 0x0000000000000000
[18:15:38] [error] At: filesystem_stdio.dll + 0x84e90
[18:15:38] [error]     Northstar.dll + 0x52df4 (0x179b2df4)
[18:15:38] [error]     ntdll.dll + 0x27fd6 (0x170027fd6)
[18:15:38] [error]     ntdll.dll + 0x60a25 (0x170060a25)
[18:15:38] [error]     ntdll.dll + 0x5e5ae (0x17005e5ae)
[18:15:38] [error]     filesystem_stdio.dll + 0x84e90 (0x1e3c4e90)
[18:15:38] [error]     filesystem_stdio.dll + 0x61df0 (0x1e3a1df0)
[18:15:38] [error]     filesystem_stdio.dll + 0x5d15c (0x1e39d15c)
[18:15:38] [error]     filesystem_stdio.dll + 0x180b8 (0x1e3580b8)
[18:15:38] [error]     filesystem_stdio.dll + 0x182a6 (0x1e3582a6)
[18:15:38] [error]     filesystem_stdio.dll + 0x18cf5 (0x1e358cf5)
[18:15:38] [error]     filesystem_stdio.dll + 0x1837a (0x1e35837a)
[18:15:38] [error]     engine.dll + 0x1516c1 (0x364a16c1)
[18:15:38] [error]     engine.dll + 0x1511f8 (0x364a11f8)
[18:15:38] [error]     engine.dll + 0x150b55 (0x364a0b55)
[18:15:38] [error]     engine.dll + 0x1346c5 (0x364846c5)
[18:15:38] [error]     engine.dll + 0x1c7d2a (0x36517d2a)
[18:15:38] [error]     launcher.dll + 0xb9d1 (0x19d6b9d1)
[18:15:38] [error]     launcher.dll + 0x15205 (0x19d75205)
[18:15:38] [error]     launcher.dll + 0x15ae9 (0x19d75ae9)
[18:15:38] [error]     launcher.dll + 0x15afd (0x19d75afd)
[18:15:38] [error]     launcher.dll + 0xd386 (0x19d6d386)
[18:15:38] [error]     NorthstarLauncher.exe + 0x48dd (0x1400048dd)
[18:15:38] [error]     NorthstarLauncher.exe + 0x8068 (0x140008068)
[18:15:38] [error]     kernel32.dll + 0x29a89 (0x7b629a89)
[18:15:38] [error]     ntdll.dll + 0x66ecc (0x170066ecc)
[18:15:38] [error]     NorthstarLauncher.exe + 0xfffffffec0000000 (0x0)
[18:15:38] [error]     NorthstarLauncher.exe + 0x80d8 (0x1400080d8)
[18:15:38] [error]     NorthstarLauncher.exe + 0x7fef0000 (0x7fef0000)
[18:15:38] [error]     NorthstarLauncher.exe + 0xfffffffec0000000 (0x0)
[18:15:38] [error] RAX: 0x5e7
[18:15:38] [error] RBX: 0x64728b0
[18:15:38] [error] RCX: 0x7d92400
[18:15:38] [error] RDX: 0xfffffffff826dc00
[18:15:38] [error] RSI: 0x1e428980
[18:15:38] [error] RDI: 0x64728b0
[18:15:38] [error] RBP: 0x1
[18:15:38] [error] RSP: 0x86e258
[18:15:38] [error] R8: 0x5e7
[18:15:38] [error] R9: 0x2f
[18:15:38] [error] R10: 0x0
[18:15:38] [error] R11: 0x7d92400
[18:15:38] [error] R12: 0x0
[18:15:38] [error] R13: 0x1e428900
[18:15:38] [error] R14: 0x0
[18:15:38] [error] R15: 0x5e7
wine: Unhandled page fault on read access to 0000000000000000 at address 000000001E3C4E90 (thread 0104), starting debugger...

Yeah, I'll check unemulated to see what it does

[ptitSeb]

Mmm, so, the program is crashing in a portion of code that is obfuscated. I guess there is still a bug somewhere in the dynarec to find. I activated BOX64_DYNAREC_SAFEFLAGS=2 and it seems to help, but it's not enough. Need to find hat bug. That might takes time (lot's of time :( )

[p0358]

If the offset from Northstar log is correct, it does appear it crashes inside of standard memmove implementation inside of filesystem_stdio.dll while trying to execute instruction mov rax, [rdx+rcx]

[ptitSeb]

Ok, So I pushed a fix for an opcode. that combined with the SafeFlags=2 make it go much further. It eventualy crashes with:

...
[21:35:47] [info] Registering Convar ns_force_melee
[21:35:47] [info] Registering ConCommand reload_mods
[21:35:47] [info] CreateInterface ENGINE VCvarQuery001
[21:35:47] [info] Enabling hook D3D11CreateDevice
[21:35:47] [info] CreateInterface ENGINE VAvi001
[21:35:47] [info] CreateInterface ENGINE VBik001
[21:35:48] [info] CreateInterface ENGINE VENGINE_LAUNCHER_API_VERSION004
[21:35:48] [info] CreateInterface ENGINE VDataCache003
[21:35:48] [info] CreateInterface ENGINE VPrecacheSystem001
d3d11: D3D11CreateDevice: initializing d3d11 stub for northstar (github.com/R2Northstar/NorthstarStubs)
Using emulated /home/seb/wine/lib/wine/x86_64-unix/opengl32.so
0158:fixme:nvapi:unimplemented_stub function 0x7f9b368 is unimplemented!
0108:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION
[21:35:48] [info] MountVPK vpk/client_frontend.bsp
0108:fixme:keyboard:LoadKeyboardLayoutW name L"00000409", flags 0, semi-stub!
Using emulated /home/seb/wine/lib/wine/x86_64-unix/winepulse.so
Using native(wrapped) libpulse.so.0
Using emulated /home/seb/wine/lib/wine/x86_64-unix/winealsa.so
Using native(wrapped) libasound.so.2
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
warning: Unknown nb_ctl request:  4
[21:35:51] [info] Enabling hook CAI_NetworkBuilder__Build
[21:35:51] [info] Enabling hook LoadAINFile
[21:35:51] [info] Registering Convar ns_ai_dumpAINfileFromLoad
[21:35:51] [info] Enabling hook CPlayerMove__RunCommand
[21:35:51] [info] Enabling hook PrintSquirrelError
[21:35:51] [info] Enabling hook ServerCreateInterface
[21:35:52] [info] Enabling hook RegisterSquirrelFunctionHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook sq_compiler_createHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook SQPrintHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook CreateNewVMHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook DestroyVMHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook ScriptCompileErrorHook<ScriptContext::SERVER>
[21:35:52] [info] Enabling hook CallScriptInitCallbackHook<ScriptContext::SERVER>
[21:35:52] [info] Registering ConCommand script
[21:35:52] [info] Enabling hook ReadUsercmd
[21:35:52] [info] Enabling hook GetEntByIndex
[21:35:52] [info] Patched AntiTamper function export "ANTITAMPER_SPOTCHECK_CODEMARKER"
[21:35:52] [info] Patched AntiTamper function export "ANTITAMPER_TESTVALUE_CODEMARKER"
[21:35:52] [info] Patched AntiTamper function export "ANTITAMPER_TRIGGER_CODEMARKER"
[21:35:52] [info] Registering Convar ns_exploitfixes_log
[21:35:52] [info] Registering Convar ns_should_log_all_clientcommands
[21:35:52] [info] Enabling hook _CServerGameDLL__OnReceivedSayTextMessage
[21:35:52] [info] CreateInterface SERVER ServerGameDLL005
[21:35:52] [info] CreateInterface SERVER ServerGameEnts002
[21:35:52] [info] CreateInterface SERVER ServerGameClients004
[21:35:52] [info] CreateInterface SERVER VServerDllSharedAppSystems001
[21:35:52] [info] CreateInterface ENGINE VTOOLFRAMEWORKVERSION002
[21:35:52] [info] Host_Init()
0108:fixme:thread:NtSetInformationThread ThreadIdealProcessor stub!
0108:fixme:thread:NtSetInformationThread ThreadIdealProcessor stub!
0108:fixme:thread:NtSetInformationThread ThreadIdealProcessor stub!
0108:fixme:thread:NtSetInformationThread ThreadIdealProcessor stub!
[21:35:52] [info] LoadPakAsync common.rpak 1
[21:35:52] [error] Northstar has crashed! a minidump has been written and exception info is available below:
[21:35:52] [error] Cause: Access Violation
Attempted to read from: 0xFFFFFFFFFFFF7FD6
[21:35:52] [error] At: server.dll + 0x1f408b
...

I looked at the disassembly for the crash, and it's an RDI=1, then R15 is set to RDI minus something, and RDI is check for NULL, but it's 1 so it goes further and crash when accessing [R15+0x14]... No sure why RDI is just 1. I'll need to debug more (probably not tonight).

If you want to try on your side, update to latest Box64, and create ~/.box64rc (or change /etc/box64.box64rc) with:

[NorthstarLauncher.exe]
BOX64_DYNAREC_SAFEFLAGS=2

So it will automatically set the flags just for the server.

[emma-miler]

Is that the entire crash log at the end, or was it cut off? Ill look into the crash a bit in any case.