Problematic dependencies

[YoshiRulz]

If it wasn't obvious, these prevent us from moving to .NET 8 (see #1415).

• [x] FlatBuffers.Core
• [x] GongShell
• [x] Microsoft.VisualBasic
• [ ] Newtonsoft.Json
  • WIP: #3932
  • ~~We have the option of replacing this with System.Text.Json at any time.~~
    • The more I learn about Newtonsoft.Json, the more I think we should replace it as soon as possible. See docs supplement.
  • As of 2021-07, System.Text.Json can use Source Generators instead of reflection. (But it could be simpler.)
• [x] NLua
• [x] OpenTK replaced in dev builds from https://github.com/TASEmulators/BizHawk/commit/78f5e755343d279f6c05821751f14cea021b7672
• [x] OpenTK.GLControl replaced in dev builds from https://github.com/TASEmulators/BizHawk/commit/78f5e755343d279f6c05821751f14cea021b7672
• [x] SlimDX replaced in dev builds from https://github.com/TASEmulators/BizHawk/commit/78f5e755343d279f6c05821751f14cea021b7672
• [x] System.Data.SQLite replaced in dev builds from https://github.com/TASEmulators/BizHawk/commit/bb96825c60fbf6900bf5fd0ad20a6ea6360acfee
• [x] System.Drawing.Common partially replaced in dev builds from https://github.com/TASEmulators/BizHawk/commit/7b644f771e4659a648de279fe56f61d60241ab8a—some more usages may be replaceable with Magick.NET, otherwise see System.Windows.Forms
• [x] System.Web
• System.Windows.Forms
  • We need to #4350 eventually. But if we want to leave that until after .NET 8, Linux will need to stay on .NET Framework and have separate "binaries" (it mostly does already). And be mindful of breaking changes in .NET Core WinForms.
• [ ] Per comments here we can't update Vortice because it dropped .NET Standard support.
  • Yoshi has a fork which downgrades its Vortice.Mathematics dep and re-adds the target, and separately a fork of that library which copy-pastes a bunch of code from the BCL in order to re-add the target. Presumably the maintainer isn't interested in either approach.

re: adopting existing libraries to replace our own code, Yoshi has some notes here. If there's enough interest in that you're welcome to open a tracking issue on GitHub instead.

[nattthebear]

Could we leave the net standard libs out of this? They're already net core compatible

I don't see us changing UI framework before .NET 5, so Windows and Linux will need separate "binaries".

We're already in the situation where the windows build and the linux build use completely different copies of system.windows.forms, yet things work with us only building binaries once. Why will that change?

[nattthebear]

Related https://github.com/google/flatbuffers/issues/4162

[YoshiRulz]

Could we leave the net standard libs out of this?

Moved out of table.

Why will [build once, run anywhere] change?

Currently, the same binary can be used on Mono and Framework on Windows because it's the same TFM (net48). Using net5;net48 will allow a binary for Windows, and a binary for Mono, both with WinForms support. net5 cannot be loaded by Mono, and Mono will never get Core 3 or .NET 5 support because the planned .NET 6 will unify the BCL implementations.

[nattthebear]

because the planned .NET 6 will unify the BCL implementations

Do you have a source for this? I've been looking and it's hard to find good information

[YoshiRulz]

[<praise of .NET Core>] .NET 5 builds on this work, taking .NET Core and the best of Mono to create a single platform that you can use for all your modern .NET code. [... .NET 5 will] expand the capabilities of .NET by taking the best of .NET Core, .NET Framework, Xamarin and Mono.

—Introducing .NET 5, 2019-05-06

Last year, we laid out our vision for one .NET and .NET 5. we said we would take .NET Core and Mono/Xamarin implementations and unify them into one base class library (BCL) and toolchain (SDK). [... Because of the pandemic,] unification will be truly completed with .NET 6, our Long-Term Support (LTS) release. Our vision hasn’t changed, but our timeline has.

—Announcing .NET 5 Preview 4 and our journey to one .NET, 2020-05-19

[YoshiRulz]

So I had a play with this thing (the Windows version works in Mono, who would have guessed) and the situation doesn't look too bad. GongShell, OpenTK 3, and SlimDX all reference System.Windows.Forms, ~~but should work in .NET 6 on Windows. IIRC natt saw that in practice when testing the hammer-and-duct-tape branch.~~ and don't seem to work in .NET 6 on Windows.

In other news, .NET 6 doesn't include WinForms for Linux, even though Mono currently supports it, and they're instead pushing ahead with MAUI (a.k.a. Xamarin.Forms). Given that, my plans for the Linux port are as follows:

• Split the binaries. This is basically done in the hammer-and-duct-tape branch. In effect, the current EmuHawk project becomes EmuHawkMono and a new EmuHawk which targets .NET 6 is created. In practice, the *Mono project files are new, and minimal changes are made to the existing project files.
• Keep OpenTK 3. I think we can use OpenTK 4 in EmuHawk and OpenTK 3 in EmuHawkMono with a bit of abstraction effort.
  • I'm assuming that OpenTK 3 and/or SlimDX works in .NET 6, otherwise OpenTK 4 obviously needs to happen before EmuHawk is retargeted.
  • The OpenTK project has since decided that gradual migration is dumb, nobody would want that, and dropped .NET Standard 2.0 -_-
• Do nothing. The Linux port can keep the ugly-but-functional status quo for a couple of years maybe, until Mono starts being dropped from distros.
  • I'm building Mono from source now, so it can be forever.
• Get tired of doing nothing. WinForms might not be getting removed from .NET, but it's also not getting any better. At the same time, all our library projects are stuck on Standard 2.0, unable to use some of the C# 8.0 through 10.0 language features. To reunify the binaries, we'll first need to decide if we want to switch to MAUI or to a third-party option (I'm aware of Avalonia, Dear ImGui, Eto, and Uno), and then focus on the one framework instead of dividing our efforts.
  • Once the Linux front-end is replaced with one that works on .NET 6, every project can target .NET 6. But, IMO, we should create a new front-end with the expectation of using it on Windows too as having two front-ends is just more maintenance work. Especially when one of them is WinForms.

edit 2022-08-12: I've just been made aware of the ActiveQt library/framework which appears to require choice of Managed C++ or COM? Not sure if Mono supports either. edit 2024-03-21: Found this thing on HN which I think is ImGui but with vectors? https://github.com/rive-app/rive-sharp

[InfamousKnight]

Opentk 4.4.0 came out like 5 days ago and mentions joystick fixes. We can't update that on Linux from my understanding of this?

[YoshiRulz]

No and that's off-topic.

[Sappharad]

• Get tired of doing nothing. WinForms might not be getting removed from .NET, but it's also not getting any better. At the same time, all our library projects are stuck on Standard 2.0, unable to use some of the C# 8.0 and 9.0 language features. To reunify the binaries, we'll first need to decide if we want to switch to MAUI or to a third-party option (I'm only aware of Dear ImGui), and then focus on the one framework instead of dividing our efforts.

The other third-party option that was recommended to me several years ago was Eto, which translates to native options on their respective operating systems (WinForms, WPF, GTK, Cocoa): https://github.com/picoe/Eto The API very much resembles that of WinForms, which made it an easy choice back then. Although it seems Eto is still getting active commits, I'm not necessarily recommending it but rather just issuing a reminder that it exists.

I had started a crappy recreation of the WinForms UI in it several years ago, only to abandon it because I didn't want to spend the time necessary to recreate everything and moved back to WinForms before mostly suspending the macOS port entirely. https://github.com/Sappharad/BizHawk/tree/EtoHawk/BizHawk.Client.EtoHawk (That last commit from 2018 is actually a problem I ran into later with WinForms and ended up solving, but I don't think it affected Windows.)

[YoshiRulz]

~~SharpCompress cannot be upgraded past 0.26.0—currently tracking as adamhathcock/sharpcompress#604.~~ fixed in ae9135807, I was overcomplicating it

---

tracking opentk/opentk#937 re: OTK3 keyboard init heisenbug; my OpenTK watchlist

[leo60228]

FlatBuffers.Core is netstandard2.0-only, from what I can tell.

[leo60228]

https://github.com/emclient/mac-playground has a port of Mono's WinForms to netstandard2.1. I'm not clear on whether it's compatible with Linux or just macOS, though.

[Sappharad]

That is just for macOS. It’s what I was using for the Mac port after Apple dropped 32-bit support. It won’t help you on Linux

[leo60228]

OpenTK.GLControl was renamed to OpenTK.WinForms, which is in beta and supports netcoreapp3.1: https://www.nuget.org/packages/OpenTK.WinForms

[jdpurcell]

Potential SlimDX replacement, targets .NET Standard 2.0: https://github.com/amerkoleci/Vortice.Windows

Supports Direct3D9, DirectInput, XInput, and XAudio2 among others, but not DirectSound. A few years ago I tried out the XAudio2 part and got it working in my forked repo: https://github.com/jdpurcell/BizHawk/blob/35ef85936666dbfdc1d6ad6bf87251f95cb51dbd/BizHawk.Client.EmuHawk/Sound/Output/XAudio2SoundOutput.cs

[YoshiRulz]

Vortice.XAudio2 doesn't seem to have a method for enumerating devices, i.e. to replace this: https://github.com/TASEmulators/BizHawk/blob/c0056123cd198ba51408a2087d57c131f95e89a3/src/BizHawk.Bizware.DirectX/XAudio2SoundOutput.cs#L27

I also noticed SharpAudio in its description which encapsulates both XAudio2 and OpenAL, ~~but that library is .NET Standard 2.1+ (see feliwir/SharpAudio#44)~~. edit: and adopted Vortice recently, so the latest beta on NuGet doesn't include it edit edit: forked it, this branch is 2 tiny commits on top of the latest beta (not on NuGet) which add .NET Standard 2.0 support, and this branch has a Nix expression which should be able to build any future beta/release w/ the same patches.

edit 2023-04: Vortice.Win32 is not suitable as a replacement for the external methods in BizHawk.Common/Win32.

[jdpurcell]

I believe that's because Vortice uses XAudio2 2.9/2.8 whereas SlimDX uses XAudio2 2.7. Microsoft stated in the changes for XAudio2 2.8:

Device enumeration functionality has been removed from XAudio2; the GetDeviceDetails and GetDeviceCount functions have been removed from the IXAudio2 interface. Apps that want to render to other audio devices on the system must pass a device identifier string to CreateMasteringVoice instead of a device index. The default audio render device can still be created without enumeration.

They don't make it clear how to get the "device identifier string" but I found a discussion where someone mentions Windows.Devices.Enumeration.DeviceInformation.FindAllAsync. I didn't want to mess with it anymore after that 😋.

I think Direct3D9 might be the bigger problem though - I did start trying to convert some of that code out of curiosity, and although I'm not very familiar with Direct3D, I'm pretty sure Vortice's implementation of D3D9 is incomplete. My impression is that the author would be unwilling to fix that himself, although would accept pull requests if someone wanted to finish the missing features.

[YoshiRulz]

I have a feeling we stumbled onto the same social.MSDN thread (login required)... edit: found another C++ code snippet that doesn't require login

We won't have any problems w/ D3D9 if we switch to its D3D12 backend instead. (And if we use Vortice via Veldrid then Vulkan is also available. Verily.)

[YoshiRulz]

This seems a good place to put this—.NET Framework 4.8.1 is real and available now, but only for Win10+, and it only includes the promised accessibility changes, not .NET Standard 2.1 support or anything "new". I don't think we should upgrade, and I'm not even going to bother checking if Mono runtime supports it.

[CasualPokePlayer]

FlatBuffers.Core ? Unknown (Emulation.Cores). IDK, natt hacked this in for Nyma. Presumably serialisation.

Seems like since https://github.com/TASEmulators/BizHawk/commit/b56e229a45446fa62026ab16aeab201b26104bf4 our copy targets NET Standard 2.0, so shouldn't actually be any issues here with that I think.

Related google/flatbuffers#4162

Seems that this too has moved along (official NuGet available now, albeit still beta), probably not really needed for this issue though?

[YoshiRulz]

Saw that. I will be switching to NuGet once it's stable, even if it's not strictly necessary.

[YoshiRulz]

@nattthebear I've pushed branch update-flatbuffers to do that ^ but I believe this change means that the C++ flatbuffers.h needs updating also. https://github.com/TASEmulators/BizHawk/blob/e87d3065718719b2abd5895886dceeb91d7f7346/waterbox/nyma/NymaTypes_generated.h#L11-L14

[nattthebear]

Seems reasonable. You should just be able to delete the contents of waterbox/nyma/common/flatbuffers/* and replace with a new version of flatbuffers, and then rebuild all the nyma cores.

[YoshiRulz]

I still can't do that on my machine. With nattthebear/musl#3 I get through most of it until:

$> ./make-all-released-cores.sh
cxx mednafen/src/error.cpp
cxx mednafen/src/VirtualFS.cpp
cxx mednafen/src/FileStream.cpp
cxx mednafen/src/MemoryStream.cpp
cxx mednafen/src/Stream.cpp
cxx mednafen/src/file.cpp
In file included from mednafen/src/mednafen.h:4,
                 from mednafen/src/error.cpp:22:
mednafen/src/types.h:58:10: fatal error: cmath: No such file or directory
   58 | #include <cmath>
      |          ^~~~~~~
In file included from mednafen/src/mednafen.h:4,
                 from mednafen/src/VirtualFS.cpp:33:
mednafen/src/types.h:58:10: fatal error: cmath: No such file or directory
   58 | #include <cmath>
      |          ^~~~~~~
compilation terminated.
[...]

I've force pushed the branch for someone else to check. ~~I only overwrote the flatbuffers headers that were present, assuming the rest were unused.~~ Added the rest to resolve below problem. Can those come from a submodule instead?

[CasualPokePlayer]

I tried building them on your branch myself and ran into a different error, with "flatbuffers/array.h" missing. Seems there's also a lot of other header files missing too.

The missing <cmath> error anyways I think is just something went wrong for you when you built the llvm/libcxx part (it should normally reside in sysroot/include/c++/v1/cmath

[YoshiRulz]

Pushed https://github.com/TASEmulators/BizHawk/commit/04fcf59afe75e7b4e938ea1ad0f10ab5f467529a and 3 prior commits.

[alkaris2]

Seems I got the issue with the System.Windows.Forms thing.

BizHawk/src/BizHawk.Client.EmuHawk/MainForm.cs(3434,39): error CS1503: Argument 2: cannot convert from 'BizHawk.Client.EmuHawk.FolderBrowserEx' to 'System.Windows.Forms.CommonDialog'

My current version of .NET on Linux is 7.0, SDK Version: 7.0.101, is there no work around for this to make it work? Can we just skip the System.Windows.Forms.CommonDialog and make it use the version in 7.0 and build anyway?

[CasualPokePlayer]

~~The error is something wrong from the most recent commit (affects Windows too). Just go back to the previous commit or wait until it's fixed.~~ This is fixed.

[CasualPokePlayer]

Something to note, OpenTK.Input 4 is actually netstandard 2.0 (the rest of OpenTK is not), so us using it for host input would be (mostly) a non-blocker (host graphics/sound is obviously still an issue).

[YoshiRulz]

There was already an option for input on Linux, evdev-sharp, which would have involved building from source. But while we're on the subject, allow me to quote myself:

OpenTK 4 and 5 are .NET Core only, though IIRC OTK 4 was .NET Standard until late in beta, maybe it can still work just by adding the target and building from source?