RetroArch
RetroArch copied to clipboard
libretro
[Netplay] Dealing with core backwards incompatibility
Description
See: https://github.com/libretro/FBNeo/issues/996
- Core can break backwards compatibility within days apart, without version increments.
- Refusal of core developers to endorse a stable repository for cores (outright threatening to cease support).
I've been unable to netplay fbneo for the past few days because the hosts are running RetroArch 1.10.3 stable but fbneo versions prior to the one that breaks backwards compatibility (nightly). These players are randoms that I just find in the public lobby, simply telling them to update the core isn't possible.
RetroArch netplay's has multiple advantages over other rollback-based services, with its main strength being its support for more than 2 players. However, with cores not guaranteeing compatibility within the same version (same version number, not same commit), RetroArch's netplay for those cores is not reliable at all.
Expected behavior
Cores should remain backwards compatible for some time, if the version isn't incremented.
Actual behavior
We only have unstable cores in the core downloader/updater, therefore, backwards compatibility is compromised on cores that get worked frequently.
Side notes
I've been working on networks since the early 2000s, and I've never worked on a project where network enabled components were distributed to the general public straight from unstable. We often had alpha and/or betas (closed and open) before we would push that update to stable and thus, to the general population. Problems (as shown here) and security leaks are two things you can expect to encounter if you don't follow with these.
@LibretroAdmin You argued a lot with me about keeping netplay backwards compatible, even when hacks were necessary to do so. But this is effectively pointless if cores can and will break it on a whim. I've been working tirelessly to improve netplay, but I'll stop if this issue is not addressed. Solutions include the already mentioned stable repository for cores, or a tag in the info files that allow me to display a very visible notification that the core is not reliable for netplay and should always be updated before hosting/joining (this will be a huge problem for platforms which can't update cores from the unstable repository).
It's also really important for netplay users to express their thoughts on this as it affects them all. RetroArch's netplay seems to always be treated as a second-class citizen and this needs to stop, or deprecate it and recommend other services.
@Chaos81 You've a whole community that netplay solely on RetroArch. You may want to express your opinion here.
Well I would suggest a new fork for netplay that the version number can be bumped. Seems the dev of this core has very little interest in the projects netplays compatibility for the userbase and that is fine,
The project as a whole needs to address this with a new unbiased fork that bumps up stable when needed if a compromise can not be met in situations like this. Either that or remove the cores netplay compatible status as it no longer fits this description for the project needs as a whole. I wouldnt throw all your improvements away for one core. Let the maintainer of this core make the choice and dont stress yourself. Someone else will no doubt make a netplay fork if the original maintainer has no interest.
Well I would suggest a new fork for netplay that the version number can be bumped. Seems the dev of this core has very little interest in the projects netplays compatibility for the userbase and that is fine,
The project as a whole needs to address this with a new unbiased fork that bumps up stable when needed if a compromise can not be met in situations like this. Either that or remove the cores netplay compatible status as it no longer fits this description for the project needs as a whole. I wouldnt throw all your improvements away for one core. Let the maintainer of this core make the choice and dont stress yourself. Someone else will no doubt make a netplay fork if the original maintainer has no interest.
It's not just one core; the way it's now, any core can break netplay backwards compatibility without any warning. It just happened to be fbneo at this time.
As for the fork, afaik, he doesn't want that either.
I wouldnt throw all your improvements away for one core. Let the maintainer of this core make the choice and dont stress yourself.
Addressing this before it leads to drama.
The reason I would stop (more like go into another hiatus) is because I can't keep netplay on a working state like this. Any improvements will be shadowed by netplay simply not working and if you think logically, this is a waste of a developer's time.
It's not just one core; the way it's now, any core can break netplay backwards compatibility without any warning. It just happened to be fbneo at this time.
As for the fork, afaik, he doesn't want that either.
Im pretty sure other cores would be willing to compromise. You are right though this wont work with some kind of versioning infrastructure. I seen this as an issue on the other thready of badly compromising backwards compatibility. I think youve have tried all your avenues in dealing with this core dev might be better talking to the libretro team rather than core dev for a big picture solution.
Regarding his views on a fork its open source/source available whats best for the project will happen regardless I would imagine. The people at the top can make a policy on the future of netplay. If this keeps up people will just use things like fightcade that do make netplay a reality.
Regarding his views on a fork its open source/source available whats best for the project will happen regardless I would imagine. The people at the top can make a policy on the future of netplay. If this keeps up people will just use things like fightcade that do make netplay a reality.
Wouldn't be any different from a stable repository for cores. They would simply stop supporting libretro aswell as the possibility of bad relations between libretro and another emulator team, which I don't want. Obviously a solution for this needs to be found because active netplay development can't continue like this.
The reason I would stop (more like go into another hiatus) is because I can't keep netplay on a working state like this. Any improvements will be shadowed by netplay simply not working and if you think logically, this is a waste of a developer's time.
I can completely understand your frustration. The simple solution you suggested is workable stable releases for netplay and nightly dont work on netplay. This will stabilise netplay for the userbase as a whole. I think you both have different goals and only one wants to compromise. I sure the other dev will see the big picture he probably feels its something he wanted but didnt think the side effects to user base as whole and feels he opinions are not validated. When its really a big picture thing. Asking devs to bump the version monthly or so is not a big ask.
Wouldn't be any different from a stable repository for cores. They would simply stop supporting libretro aswell as the possibility of bad relations between libretro and another emulator team, which I don't want. Obviously a solution for this needs to be found because active netplay development can't continue like this.
Well this is true incompatible cores with no versioning just wont work. If devs cant update something as simple as a version on changes that effect netplay it might be worth removing these cores from netplay capable status and just keep the ones that make netplay a priority work in the lobbys. That way at least is doesn't give the impression of netplay is broken and devs can do what they please with the core as far as supporting netplay goes.
Some platforms don't really have the luxury of doing that. For arcade, it's either fbneo or fbalpha, if we remove the netplay compatibility status from fbneo, the only option for arcade is fbalpha, which is severely outdated.
Tagging cores as not netplay reliable but not outright locking them out of netplay is better than simply removing those cores from netplay, although as said before, platforms which can't use the core downloader/updater will be in trouble, just as it's right now.
The biggest problem I see with any other solution than a stable version to work from is people will have problems. Example if you use commit hashes you cant join servers with different hashes. If the person hosting an older hash than you how do you get that version.
Without a baseline to work from it just wont work long term and users will get the impression its broke like they do now. Since we cant get a baseline with this core per say the version rarely changes its not a great candidate for netplay support. A certain critera should be set if you want to support net play claims. I cant see netplay going forward without some intervention on implementation conditions to support it.
I also dont think this is something you can fix it has to be a platform/team choice its not fair for this to fall on your lap with every dev.
This issue goes beyond netplay though seems savestates for this core will be unstable for users. Unless it checks the size and deletes any incompatible ones to stop any issues occurring.
I also dont think this is something you can fix it has to be a platform/team choice its not fair for this to fall on your lap with every dev.
Aye, which is why I've created this issue as I can't solve it myself. It falls on either libretro's infrastructure (core downloader/updater) or the cores themselves, both of which I've no saying in the matter.
Well hopefully something fruitful comes of this in the long run. Either way you will have your answer in time . I think for this core in question and the devs feelings on the matter it has no future with netplay in a workable way for the user base. Hopefully other core maintainers see things differently and the platform can let people get a better better experience from netplay.
I already explained it all in the other issue :
- those "backward compatibility breaking updates" are not "on a whim", they are fixing issues including netplay ones.
- freezing versions still doesn't guarantee the user will use the latest freezed version
- freezing versions is turning support into a painful experience for both devs (at the very least, those like us who have been cooperating with the libretro project and providing fixes on a daily basis) and users
- expecting us to increment versioning every time there is a meaningful commit for netplay is an unrealistic burden, because we make several commits a day and most of them are likely to break netplay backward compatibility on a subset of the 18000+ games we support.
outright threatening to cease support
As a matter of fact, we can't provide support if a broken version of our emulator is frozen in time on libretro's side. This is the direct consequence of the policy change you are requesting. Hanging around libretro github/reddit/discord/forum just to tell users "thanks for the report, this is a bug on our side, we will fix it asap but you won't be able to download the fixed core anytime soon without first messing around with your retroarch.cfg file" just doesn't feel right to me.
It's also really important for netplay users to express their thoughts on this as it affects them all.
Your proposition also affects all non-netplay users so maybe you should start caring about them.
Simply said, this "frozen core repo" solution you are proposing is causing major issues without actually guaranteeing the original problem of mismatching core versions is solved. You should find a better solution, preferably one that only affect netplay. The only one that comes to mind would be to enforce client-side to download and use some tmp core which is the exact same version as host-side, that'd require major changes in retroarch and some storage to archive cores.
As for the fork, afaik, he doesn't want that either.
I actually couldn't care less at the condition this fork doesn't become a burden for our team, meaning it must not be named in a way that would bring confusion and redirect all support to our team while it's actually not maintained by us.
Ofc, i'd need to confirm with the other members of the team if they are ok with this.
How exactly is having two different repos "affecting" other users? If you don't want to use the stable repo, use the nightly, It's literally a setting within RetroArch called "Buildbot Cores URL". But that's literally your argument for every single comment you make.
Quoting this again for emphasis:
I've been working on networks since the early 2000s, and I've never worked on a project where network enabled components were distributed to the general public straight from unstable. We often had alpha and/or betas (closed and open) before we would push that update to stable and thus, to the general population. Problems (as shown here) and security leaks are two things you can expect to encounter if you don't follow with these.
On a whim means a sudden decision, which applies to breaking savestate support for older versions without incrementing the version number.
And this passive-aggressive condescending attitude is getting old. I just saw how you treated @MistyDreams at the other issue (and this is the second time I've seen you treating him like that).
I already explained in length, in this issue and the other, why this "frozen core repo" actually doesn't really solve anything, and why incrementing version in the way you want is unrealistic and wouldn't be helpful.
If you don't want to use the stable repo, use the nightly
This "frozen core repo" (stop calling it stable, it's not) is a trick only helpful for your netplay mismatch issues, and shouldn't become the default repository for downloading cores.
And this passive-aggressive condescending attitude is getting old. I just saw how you treated @MistyDreams at the other issue (and this is the second time I've seen you treating him like that).
We have worked for years on this project, willingly improving netplay support along the line, building a user base who trusts us in improving our emulator on a daily basis. And now some people come and say "you don't care enough about netplay, everything should be centered around netplay, by default we'll freeze versions for everyone because it might serve the needs of our netplay community, and if your users aren't happy with that you can always tell them to change the default configuration". Sure, i'm totally the person in the wrong here. I'll stop there, it doesn't seem you can understand what i'm explaining.
No need to get heated. We're all acting in good faith, and we all want everything to work as effectively and seamlessly as possible.
We already have a solution to this issue, which is: stable releases with core snapshots for those releases: http://buildbot.libretro.com/stable/1.10.3/windows/x86_64/RetroArch_cores.7z
Those people for whom netplay is their top priority can use these stable snapshots, as they will never change, and they should match the stable releases for other platforms for cross-platform play.
No need to get heated. We're all acting in good faith, and we all want everything to work as effectively and seamlessly as possible.
We already have a solution to this issue, which is: stable releases with core snapshots for those releases: http://buildbot.libretro.com/stable/1.10.3/windows/x86_64/RetroArch_cores.7z
Those people for whom netplay is their top priority can use these stable snapshots, as they will never change, and they should match the stable releases for other platforms for cross-platform play.
- That doesn't do anything for platforms with the core updater, as the core updater will always download from nightly, even from a fresh stable install.
- They aren't the same as the one at the time of a stable release either. Check the dates between the release post here https://www.libretro.com/index.php/retroarch-1-10-3-release/ and the timestamps at the buildbot.
And I don't mean to be rude, but if the fix was that simple, I wouldn't have opened this issue. Which is also important to note, as have already been stated in the OP, there is no decent versoning here, the commits hashes are worthless for the vast majority of users.
TL;DR: netplay with currently wip cores is not reliable at all and this isn't a problem that can be fixed through code. Forcing core downloads won't work for platforms which don't allow it, like Steam.
Guaranteening that cores on the same stable version be compatible on multiple devices, should be standard imo, but some people don't seem to understand the need for standardization and the consequences for lacking it.
It's also ironic that I got a lot of shit from some contributors for wanting to break backwards compatibility between 1.9.X and 1.10.X to avoid shitty hacks and some inherently problems with supporting old broken code; but apparently, this (which can break backwards compatibility every few days) gets a pass... infuriating.
Let's please find a way to compromise here so that neither side here gets alienated and neither side leaves development. We cannot afford any developers to leave over this. We have to find a compromise if only for the sake of the community and the users. Let's please find common ground here.
@Cthulhu-throwaway We cannot force a particular way of doing things suddenly on the FB Neo team. We have to find a compromise.
I'll repeat again - throwing down ultimatums that one leaves development over a design decision is not productive on either side, and it is unfair and impossible for any of us to start showing favoritism to either side. If something is really that disruptive, the idea should be dropped, because coding decisions in no way can result in crucial developers having to leave or butt heads. We have to compromise then to let peace prevail. There is no point in chasing perfection if it's going to kill off a good amount of contributors in our current ecosystem. This is a team-based effort and decisions have to be made that the majority can agree on and can work with. Alienating people is not the solution.
For what it's worth, I don't really like the idea of some 'netplay core repository' or whatever is being suggested here. I'd suggest we stay with what we currently have and we don't try to overegg the pudding here unnecessarily. It really is not that serious to warrant butting heads with one another.
you have done excellent work and nobody is denying this. Let's take a step back though and not create a huge issue here unnecessarily when there doesn't need to be. We are seriously exaggerating any actual issue here.
We already have a solution to this issue, which is: stable releases with core snapshots for those releases: http://buildbot.libretro.com/stable/1.10.3/windows/x86_64/RetroArch_cores.7z
I agree with this, this is more than enough. We really don't need to resort to some controversial solution for nightlies and then risk developers from either side getting upset over it. If it's that controversial then we should not pursue it. Furthermore, the team is already overemburdened with enough stuff to do, and having to add even more to our workload to setup new workflows is not really something that is currently in our interest. If anything, we want less technical debt.
These players are randoms that I just find in the public lobby, simply telling them to update the core isn't possible.
Honestly a far better solution is just having something so that before a netplay session is started, if internet connectivity is available, it checks the version, and if it's out of date, it attempts to just download the newest core. It should be easy to just fire off a task for that. That is far more productive than creating some 'stable netplay core version' repository or whatever is suggested here. This is way too overcomplicated and we really cannot be asked to maintain all of that, it is yet even more laborous work and we already have way too much technical debt as is.
From what I hear, @barbudreadmon would be down for this solution and it would solve the entire impasse right now.
Let's not needlessly overcomplicate things here. The best-case solution if a version is out of date is to simply update the core, period. There is not enough manpower or labor force in this project to have to maintain lists of 'stable' versions for netplay, people are assumed to either be on stable core versions or on the latest nightly cores. Anything else is going to be a total maintenance disaster/nightmare and something that there simply is no manpower or labor power for.
or a tag in the info files that allow me to display a very visible notification that the core is not reliable for netplay and should always be updated before hosting/joining (this will be a huge problem for platforms which can't update cores from the unstable repository).
I'd suggest instead of creating a scary warning message, have the tag in the info file, HOWEVER, instead of doing a scary warning, have it automatically update the core on the spot then for platforms where this is possible (should be possible for both statically linked and dynamically linked targets I think when online updater facilities are available). It should be as easy as triggering a HTTP download task and then extracting the core to the appropriate directory, there's already functionality for this existing. You'll have to come up with something else for Steam though which will require you to talk to @Mats since he runs the Mist integration part of the Steam version.
Anyway, I'm afraid all of this is going to result in tons more work either way. I'd suggest we hold our horses right now, it really is not much to ask users to just make sure they are on the same core version. That should be expected and there is a handy and easy 'update all cores' button exactly for this purpose.
Dude, it's not ambition, it's literally in the first post that I can't join 1.10.3 (latest stable) rooms running fbneo because I am running a newer commit of fbneo that isn't backwards compatible with something as short as 2 months, but is the only one available through the core downloader. I know how to downgrade and NOW I know what's causing my issues joining them, but do you really expect the majority of your users to go through all of this to know they just got screwed by backwards incompatibility changes that didn't increment the version of the core and is pushed silently into the core downloader?
And it's not an ultimatum; until now I assumed those cores remained backwards compatible on the same version number, now I know better and I can't continue working on netplay until this is solved. I am finishing my work on implementing the client bans and fixing an input issue (since someone from Snes9x recently reported in to my request for information), but I'll be effectively on a hiatus until this is resolved in a satisfactory fashion, might even suggest deprecating netplay if this isn't the case because I doubt any other developer will stay long as soon as he finds out netplay gets broken every now and then because of changes outside of his reach.
Also, it's not me that needs to compromise, I can't do anything; this falls beyond the netplay code in RetroArch as stated multiple times.
Already went through the core download issues on platforms that won't allow it. So, not much else for me to say other than to read my previous comments carefully.
And again, and you know this very well, as you were one of the people giving me a lot of shit for a single break in backwards compatibility, how come this gets a pass just like that? Sure, you can't tell them what to do with their own core, but you can make better decisions than them with your infrastructure.
Finally, I am not just the current main contributor to netplay, but also an active netplayer; people will often find me joining their rooms. I know most of the pros and cons of the system by now and if I am saying I can't continue working like this, it means it's a huge issue that I can't fix on my end. You know very well how discouraged I was to continue working on netplay, as we talked about this a couple of times, and this is another thing to add to the pile of not wanting to work on netplay anymore (too much work, too many problems... TOO MUCH DRAMA).
And again, and you know this very well, as you were one of the people giving me a lot of shit for a single break in backwards compatibility, how come this gets a pass just like that? Sure, you can't tell them what to do with their own core, but you can make better decisions than them with your infrastructure.
Because FB Neo developers would leave over this (we were told). And conflict and drama over what? Let's just compromise. It is really not all that serious. It simply isn't possible for them to maintain that kind of compatibility over months. It is a project worked on by many people, they don't all have the same interests and only a handful of devs are involved in the libretro core side of things like barbudreadmon. The only workable approach there is just updating the core on both ends to the very latest version. That is the only workable solution for them there, and it's understandable.
Would also encourage you to please come on Discord so we can talk there. Things tend to get too heated on Github and it is really not productive to getting these impasses solved. Already suggested earlier meeting up there since there is still the relay issue. You can contact 'BoardsOfCanada' or 'hunterk' there.
There doesn't need to be all this drama over something that ultimately we can find a compromised solution for that doesn't involve having any developers leave, and we can just be on our merry way continuing development as usual. We prefer if we could resolve this on Discord in DMs. This is not the proper venue to solve this.
There is another bad decision, which is centralizing all of your communications on a platform like Discord. I am not going back there, man. The relay issue is just a matter of trying again later, nothing I can do about, and this one, well, nothing I can do about either, it's something you should handle. I don't handle your infrastructure and I don't contribute to cores, again, I can't do anything. This issue was opened in order to bring attention to this situation and that's a major roadblock for any netplay developer.
Also, I didn't complain about them having to maintain backwards compatibility, but rather their lack of version increment on such occasions and such commits getting pushed to the core downloader automatically, without an alternative that doesn't involve non-standard approachs (aka going to the buildbot url and download cores manually).
How I am supposed to keep netplay functional in an environment like this?
@Chaos81 You've a whole community that netplay solely on RetroArch. You may want to express your opinion here.
Luckily for me, I make the community download a specific RA package with certain features disabled, like the core updater/downloader. This way I know everyone is on the same core version. And I don't usually update the RA version unless there are some netplay or other improvements that benefit us, as it's a pain to get everyone to update.
Unfortunately, you can't do something like that with something so open to everyone. So you are going to run into these issues. The warning messages that pop up about core version mismatch are usually ignored. What about forcing a download of the matching core on connect? For those systems that can't download a core on connect, maybe adding a way to search the lobby for games with matching cores (or compatible core versions, which would require a list of versions compatible and might be too much work for developers)?
Also, like to note Cthulhu has done an amazing job on netplay. It's been working better than ever.
