firefox-devtools
Stop supporting backward compatibility until we have a technical solution to support it
After working on remote debugging for a few months, I have started paying much more attention to backward compatibility breakage than before. We already had RFC #51 to discuss backward compatibility issues, where we decided to reduce our backward compatibility policy to supporting only N to N-1 or N to N-2 setups.
However even if this is in theory easier for us, we still break backwards compatibility very often. We have no tests for backward compatibility, so unless you actively think about everything that could break it, we usually regress it without knowing.
I think it's not reasonable to keep putting this pressure and workload on the team. I propose we stop supporting any backward compatibility until we have a real technical solution to support it. I think today, our attempt to support backward compatibility results in a lot of time loss for development, and bad user experience since despite our efforts, things break frequently.
In practice, this means about:debugging would refuse connecting to older servers, and would prompt the user to download the appropriate version of Firefox instead. We can keep a preference that would still allow users to bypass this. For Nightlies with different build dates, we can probably simply display a warning with a suggestion to use matching Nightly versions.
This also means we stop actively trying to code for backward compatibility, we don't block reviews if backwards compatibility is broken.
Some options to reintroduce it below (just adding them for context, the goal is not to discuss them right now):
- manual recurring test campaign (weekly). Pros: we can set it up quite quickly. Cons: recurring costs will be high, will only catch the bigger issues (blank panels for instance)
- automated tests. We add a new task running on continuous integration, that downloads an old firefox binary and runs remote debugging tests against it. Pros: not costly once setup, can be expanded to cover new issues as we spot them. Cons: might be hard to setup the first time, will not simplify the actual support for backward compatibility in the code
- run the client N for server N: we could either ship the client with the server and retrieve it when we connect to the server, or maybe download it from a repository. We would still run the risk that a given client is not compatible with the non-devtools code, but our client folder uses very little in terms of non-devtools APIs, so it should be ok. Pros: no longer need to code for backward compat. Cons: probably the most complicated solution to put in place.
And to give some more context about the usage of backward compatibility, you can look at the following data compiled from WebIDE's usage: https://docs.google.com/spreadsheets/d/19u57uS5PcnTleCwaLp3va0YDkn65LFoQmORB2uDoBxc/edit#gid=211432069 as well as https://bugzilla.mozilla.org/show_bug.cgi?id=1528219#c13
The main info here is that Nightly users mostly debug the same version (N -> N), but DevEdition users are heavy users of backward compat (~60% N->N-1, ~15% N->N-2). So doing this change will definitely have an impact on our DevEdition users who are used to the convenience of using DevEdition to debug Firefox for Android Release. But I think it's worth doing if we can migrate them to another approach (ie download another Firefox version) and make the overall experience less buggy.
In practice, this means about:debugging would refuse connecting to older servers, and would prompt the user to download the appropriate version of Firefox instead. We can keep a preference that would still allow users to bypass this. For Nightlies with different build dates, we can probably simply display a warning with a suggestion to use matching Nightly versions.
Note that Firefox for Android (aka Fennec) Nightlies are now built out of the beta branch, and super soon (I think it's later today) they'll be built out of the esr68 branch. So you'll need to account for this.
Also do we currently have a strategy when connecting to a newer server (aka forward compatibility)?
I think my preference would be to still support connecting to the current Release from a current Nightly. As a user it makes things easier in general, even if we don't use it often.
To be a bit clearer about what I have in mind: I don't mind if we break backward compatibility for a good reason (eg: the fission work), but is it necessary to make it a rule that we don't care about backward compatibility in general? Maybe it's good enough to decide that we can break backward compatibility but that most of the time we support it for some versions?
Doing this change will definitely have an impact on our DevEdition users who are used to the convenience of using DevEdition to debug Firefox for Android Release. But I think it's worth doing if we can migrate them to another approach (ie download another Firefox version) and make the overall experience less buggy.
I'm not against this at all. If the UX for this is nice and easy for people to understand and easy for people to act on it (i.e. download the right version). Especially that it's now easy to run several versions of Firefox side by side.
It does sound like a product decision overall though, since this would be a workflow change for a large % of our DevEdition audience. It would be good to describe a bit more how unreasonable the current situation is. Is this a matter of each of us doing a bit more testing and being a bit more aware. Or is the load on the team just not bearable?
run the client N for server N: we could either ship the client with the server and retrieve it when we connect to the server, or maybe download it from a repository. We would still run the risk that a given client is not compatible with the non-devtools code, but our client folder uses very little in terms of non-devtools APIs, so it should be ok. Pros: no longer need to code for backward compat. Cons: probably the most complicated solution to put in place.
I would love for us to move to this solution. We have been dreaming about something like this for a long time, never really putting any efforts in trying it out. It sounds hard, but we've never actually looked, maybe there's a chance it's not that hard. If we still limited it to the 2 versions range, maybe there's a way for us to copy the N-1 and N-2 client code every time after a merge into 2 new client folders, and use this when establishing a connection with an older server. To avoid having to actually download the code.
As you said, we'd still have risks that an older client code doesn't run well on a newer Firefox, but that sounds very unlikely. Although this wouldn't be a 0 risk situation, so we would still be facing compatibility issues at the client|platform boundary.
Thinking out loud: what if Firefox N doesn't support XUL windows anymore at some stage, and our client code N-2 still has a few XUL windows because we only removed them in N-1?
What if const variables visibility change again in the future?
These are just random examples. I think overall compat issues would be rare, but when they do occur, they might be harder to deal with.
Maybe it's good enough to decide that we can break backward compatibility but that most of the time we support it for some versions?
Waiting for more answers before going into details, but I think it's important to say that this is what are doing today IMO. We are backward compatible by default, and allow exceptional non backward releases (eg fission non backward tracking in https://bugzilla.mozilla.org/show_bug.cgi?id=1528766)
I think it's much better to have a little user pain that is explicit (use the correct client and server versions) rather than the subtly broken version of having a maybe working remote debugging experience. I think it is really bad to have an untested code path that is hard to get right, that frequently breaks. This means that our end users have a potentially really bad experience. As project maintainers we also waste a lot of time having to fix hard to reproduce bugs.
I think we should have a really strong promise to our users to ship correct code, and the current situation doesn't live up to that promise. In summary I think it's a good idea to have a little upfront end-user frustration with matching up client/server versions, but with guarantees of a functioning product.
The ideal case would be to get a process in place to both be backwards compatible, and correct, which is enforced through testing.
I agree with @gregtatum. I think the burden is the user in any case -- if the version is subtly broken and they do not know why -- this is not a great scenario. I think however that having a process that is backwards compatible and enforced by testing is very labor intensive. We would need to set aside dedicated time for this.
That said, i think making it explicit to a user that things can break if you debug between version and if its broken go use this other thing is already a major improvement. It is small enough that we might be able to do that quickly, and explore a more complete solution over time.
A few consideration for this discussion
- Fennec will probably get an ESR treatment (while new new GeckoView based browser is being build out) and be pinned to the next Gecko ESR version.
- Since 68 (to be verified) downloading an older version of Firefox will not downgrade a profile, so users are save. Older versions are still susceptible to borked profile data due to down and upgrading and need to be treated carefully.
- Downloading older versions of Firefox is not easy, as they override the currently installed version and auto-update
- Firefox Desktop and Mobile are not necessarily moving in lockstep
- An option would be to hand-curate a compatible version, so Nightly doesn't always yell at users to use the same version. One way might be incrementing a protocol version when a major (in its semver sense) change is being made in the protocol (like a new method being added)
- Long term (as in post-Fission) we need to create a solution that gives us the freedom to iterate on the protocol with confidence. Decoupling the frontend would allow us to match the right frontend version with any backend; but needs a lot more exploration. There are probably more benefits here to explore.
