w3process
w3process copied to clipboard
w3c
Single implementations and interoperability: showing implementation experience with fewer implementations
At a recent AB meeting in which we looked at formal objections (https://www.w3.org/2021/04/15-ab-minutes.html#t03) the issue of monoculture came up. Some think that the term monoculture is not a good description so we are calling this issue Single implementations and interoperability (SII).
Roughly, SII or monoculture refers to a situation where you have a reduced set of independent implementors. That could be because everyone uses the same open source library or because there are just fewer browser engines.
We should define what constitutes sufficient implementation experience (https://www.w3.org/2020/Process-20200915/#implementation-experience) for SII situations.
(Apologies to those that do not have Member access to the above listed AB minutes, but I believe that the issue can be understood without looking at those minutes.)
"Monoculture" is more of a metaphor than a concrete description of a problem. I suggest the first step is to identify the VALUES potentially compromised by a spec driven to Rec by one implementer community and the THREAT MODEL for how those values could be compromised (and mitigated).
I recall participating in lots of AB discussion on this topic over the years; I can't find it so I guess it was unarchived email from my Microsoft account or archived discussion behind the AB confidentiality wall :-( . As best I can recall/rethink:
The VALUES certainly include:
- Utility -- it has to solve a real problem for real users and will get deployed on real websites
- Implementability - It has to be implementable in principle by other projects or vendors. At a bare minimum, other stakeholders should have to attest that they can and may implement a spec for it to advance.
- A11Y, I18N, Security (of course)
- Fairness -- advancing the spec should not INTRINSICALLY favor the business interests of the proposing community. Yeah, economies of scale and Metcalfe's law favor the large and powerful, I don't think W3c can do much about that.
- Privacy (not going to dig into that hole at the moment, but it needs to be excavated or "privacy!" will be an all-purpose spec-blocking play)
The THREATS would include:
- Utility is impacted if it is a solution from one community looking for a problem to attach it to, and using Recommendation status to promote their solution.
- Implementability is impacted if a spec is heavily tied into the architecture or API ecosystem of the single implementer.
- Fairness is impacted if it is ONLY valuable to someone with a popular search engine, ad or social network, walled garden, etc. to have the spec become a Rec
- Privacy (a bottomless pit of threats that I'm not qualified to sort out, but need to be if the Formal Objection and resolution process is to be anything other than purely political.)
This is all off the top of my head, meant to structure the discussion not provide an answer
Indeed, can we NOT use the term "monoculture" to describe this issue? One shared implementation with a very diverse community is not a monoculture, and indeed, to those outside that community, it is reductive in a dismissive way. However, there still are a set of real problems that arise when there is only one set of implementation CODE, and I do want to dive into those issues.
I would prefer to frame this as "Single implementations and interoperability: showing implementation experience with fewer implementations" or the like.
I would prefer to frame this as "Single implementations and interoperability: showing implementation experience with fewer implementations" or the like.
Done.
The threats to the Security value from single implementations are the most prominent examples of problems (e.g. SSH, https://www.zdnet.com/article/serious-security-flaw-found-in-openssh-puts-private-keys-at-risk/). Mitigation needs to be fleshed out in the Process or guidelines the "Director" uses to assess implementation experience. The AB / Process CG / Team should consult with recognized security experts while drafting guidelines. Some possible mitigations (from someone with little security expertise, so again these are discussion starters not proposed language) might be:
- A higher bar for security review of a spec with only one implementation. WG's MUST ensure that real experts review the spec and work with the open source implementer to ensure that real experts review the code.
- Some sorts of prominent warning in the /PR version of the spec at all maturity stages that there is only one known implementer and users should be more cautious than usual.
- Mandating that the WG and/or implementers have a maintenance plan in place for the spec and code so there is someone actually responsible for fixing any vulnerabilities when they are discovered.
- Mandating that the WG/implementers/maintainers subscribe to reputable vulnerability alert service to ensure they become aware when vulnerabilities are found.
Hmm, the "mandatory maintenance plan" might not be a bad thing to require of ALL Recommendations ;-)
Good thoughts, Mike. A few related things:
- The one concern I have is that new implementations appearing shouldn't require a re-issue of the spec to remove that warning. Perhaps this would be metadata triggering the styling.
- Mandates will be difficult to enforce, when a WG scatters to the winds when its charter runs out. :)
- Implementers typically are the first to hear about vulnerabilities. Perhaps what we really need to do is ensure the notifications flow in the opposite direction - that implementers explicitly commit to come back and update the spec when vulnerabilities are found, not just patch their implementation.
Good point; W3C can mandate that a maintenance plan exists on paper, it can’t force the WG members to stay engaged and follow it.
An ongoing liaison between the W3C staff and the implementing project might be more effective. And if the project loses energy or forks maybe the team can rescuing the Rec, realign with a fork, whatever
When a spec / implementation / tests are written by the same party, it's very hard to be sure that the spec is sufficiently precise to enable another implementation to be created later and actually be compatible. It might be, but it could also be somewhat vague or ambiguous and people have just failed to notice. I think we should be more stringent on the existence (and thoroughness) of third party reviews when there are few (and especially only one) implementer. Even more so if the spec editor / test author is the same party.
Another risk is implicit bias in the design, that builds upon assumptions that are true in one particular implementation, but may not hold elsewhere. Maybe we could add a self-review questionnaire to try and get people thinking about what assumptions they might have unconsciously made (serial vs parallel, sync vs async, cpu vs gpu, screen vs paper vs VR glasses…), but it's going to be very hard to replace the insights gained from an actually diverse ecosystem of implementations and deployments.
When considering what constitutes implementation experience, we should perhaps look beyond browser engines where applicable, especially given the waning diversity in the browser engine space.
For example, Deno (https://deno.land/), a modern runtime for JavaScript (and TypeScript) runs in its CI a number of web-platform-tests for W3C and WHATWG Web APIs (see https://wpt.deno.land/) and could be considered an implementation.
I feel this is an opportunity for W3C to broaden its perspective, increase diversity of implementation experience, and give a voice to a broader group of implementers.
@lucacasonato from Deno core team was happy to join efforts that could strengthen collaboration in this space.
Cc @dontcallmedom
@anssiko as far as I am aware there is no limitation that constrains implementation experience to browser engines today, and there never has been. In some WGs (like, ahem, the one I Chair), it's the exception that the implementation experience comes from browser engines, rather than the rule.
It may well be a good idea to actively promote broadening out the implementation experience, even if it is essentially a marketing and communications exercise. I don't think there's any process change needed to do that though.
Thank you @nigelmegitt for your perspective! Timed Text Working Group is a great example of a group where there already is diversity of implementers and this broad experience is considered. This may not be the case with all WGs.
To clarify, my comment was specifically about web specs that many might have considered browser-only in the past, but have now gained implementations also outside browsers. Examples include File API, Web Crypto API, WebIDL, DOM, User Timing, Web Storage, and so on that Deno currently implements. Even WebGPU API is being experimented with in Deno.
I have a feeling we're lacking adequate representation from these new implementers in many key groups working on web specs, so I wanted to bring this topic to the attention of the Process CG -- I felt this data point might be relevant to this issue regardless of whether any process changes are needed to make a change for better.
Feel free to redirect this to an appropriate place and loop in @lucacasonato.
I'd like to call attention to the proposed TTWG Charter's success criteria that seek AC approval for a more specific definition of the requirements for advancing Rec track documents than that in the Process - I think there are prior examples of specs advancing with one implementation (I'm told that WOFF2, apparently, did this, but I can't clearly see that from the Implementation Report) it is not the first time that a spec has advanced with only a single implementation.
As far as I know this is the first time that a Charter has explicitly stated that two "factors of verification" are needed, only one of which need be an implementation.
Very happy to discuss this further - either with folk interested in this or with AC Reps who might very reasonably have concerns.
CC @gkatsev
some related background on single-implementation of specs https://github.com/w3c/oss-relations/blob/main/single-oss-implementations.md https://github.com/w3c/oss-relations/issues
Has the objective of demonstrating implementation experience fundamentally changed? The current process document states:
Implementation experience is required to show that a specification is sufficiently clear, complete, and relevant to market needs, to ensure that independent interoperable implementations of each feature of the specification will be realized.
I see multiple ways of satisfying this object, depending on the domain of application at a given point in time. For example,
- a first domain of application might be satisfied with multiple proprietary independent implementations
- a second domain of application might be satisfied with multiple independent parties contributing to the same open source implementation
- a third domain of application might be satisfied with two independent implementations, one authoring and one playback
- a fourth domain of application might be satisfied with two independent playback implementations
- etc.
I would therefore argue that the process document ought to remain flexible, and continue to merely require "demonstrating implementation experience".
Has the objective of demonstrating implementation experience fundamentally changed?
What has changed is:
- There is no engaged, expert, neutral Director to make the judgments the Process expects the Director to make
- The web platform has gotten so complex that few companies can afford to implement it, so even fierce business competitors such as Google and Microsoft collaborate on a shared open source implementation. The process didn't anticipate that reality, or evolve to find a resolution to this issue to mitigate it.
- W3C has evolved to standardize technologies such as TTML that aren't really on the web in practice, but are useful in the (mostly open source) toolchains that generate web sites and services. So web platform / browser engine implementers don't participate in working groups that create the not-really-on-the-web specs.
Remaining flexible is fine in principle, but without a Director these judgment calls will become arbitrary or political. Until the Process gives better guidance, it's critical for charters to state success criteria that promote the values and mitigate the threats from single-implementation specs noted in previous comments on this issue.
@michaelchampion It sounds like the objective ("demonstrating implementation experience") has not changed, but the meant by which the objective is achieved and who determines if it has been achieved. Did I get this right?
@michaelchampion re https://github.com/w3c/w3process/issues/522#issuecomment-1050236655 :
I'm struck that you're on the one hand saying that there is no Director, and on the other apparently dismissing the idea that technologies such as TTML, and presumably many others, are part of the web, despite the fact that the (less engaged than he used to be) Director approved those technologies as W3C Recommendations many times.
Your call that these technologies are "not-really-on-the-web" is itself arbitrary and political, and looks like a manifestation of the world view that the web is only what browsers do. I'm not sure if that's the meaning you intended. I need probably not mention that TTML is a technology in use in a large number of devices that are on the web, but not browsers in the traditional sense, and from that perspective, it is the browsers that are out of step: the criterion for "on the web" is not and never has been "implemented in browser engines".
... it's critical for charters to state success criteria that promote the values and mitigate the threats from single-implementation specs ...
I fully agree with this.
. I need probably not mention that TTML is a technology in use in a large number of devices that are on the web, but not browsers in the traditional sense
Well, as a retired browser developer, I confess to thinking that "on the web" essentially means "usable with a web browser" (or "should be implemented in Chromium because real web developers have to use a bunch of Javascript to meet a common user need") but I respect that others have a broader definition.
But I would appreciate being educated on what devices consume TTML directly from HTTP URLs, which is a the broadest definition of "on the web" I can think of. Perhaps my understanding that TTML is mainly used in toolchains that generate content for the web but but seldom served directly is mistaken or out-of-date.
But TTML is a bit tangential to this issue; my Process concern is that a) there is effectively no neutral, expert, engaged Director to interpret the implementation experience requirement in a particular context; b) increasingly there is only one implementation of W3C specs; c) so the process and/or charters need to be much more explicit about success criteria for single-implementation specs so the question of whether they have been met is not arbitrary/political.
Aside: devices that consume TTML from URLs
Just to name a few:
- All HbbTV 2.0 connected TV devices/set top boxes provided with an appropriate MPEG DASH MPD
- All ATSC 3.0 devices
- Devices conforming to the Nordic profile of DVB, which incorporates the DVB TTML specification requirements
- Most BBC iPlayer and Netflix applications (to name but two that I'm certain of)
- Apple HLS implementations when given CMAF caption chunks in IMSC
I'd ask "why" again, and @michaelchampion your list at https://github.com/w3c/w3process/issues/522#issuecomment-828594305 is a good start, when you list some of the values and threats.
To that, I would add specification quality: there's a value to specifications that are easy to understand and implement; in single implementation scenarios there's an elevated risk that ambiguities are not noticed because one implementer is happy with their one implementation which they believe meets the specification text and there's no counter-view that the text can be interpreted a different way, where the difference could lead to interoperability problems.
Another risk is that work on the single implementation stops for whatever reason and it becomes stale, and then the community realises that the implementation was more important than the specification, which could be a roadblock to continued usage.
I do think that more developer eyes on a single implementation can mitigate those risks to some extent, especially when that implementation is open source.
We could consider how to describe the minimum license requirements that are needed to allow an implementation to be considered open source for the purpose of mitigating single implementation risks.
The W3C publishes Recommendations on web technologies, and we have long realized that we have technologies that are not expected to be, or needed to be, directly implemented in browsers or might be relevant to some webs, or places where web techs are used, but not the public world-wide web. I think this is a red herring here.
The question of how we demonstrate that a Rec is well-enough written to result in implementations, from the spec., that are also interoperable, is a difficult and urgent one. We have both engine diversity, increasing use of common polyfills, and increasing use of open source, to consider. (But remember Heartbleed...)
By the way, I think it's fine for a spec. to hang at CR waiting. The IETF has most of its work hanging at RFC…
Thanks for the TTML education :-)
Given that the Process is not going to change quickly, especially since the AB is distracted by the governance/funding/hosting critical situation, it's important that the considerations you list be baked into the TTWG charter.
For example, the ambiguities in CSS 2.0 (15 years ago) were addressed because a new implementer (Microsoft, back in the day they couldn't look at open source code for supposed legal reasons) working from the spec discovered that their implementation didn't interoperate with others'. That led to a massive testing effort that identified and clarified (in CSS 2.1) the ambiguities and massively improved interoperability. How could something like that work in a single-implementation environment? The same principles of independent parties working from the spec to write tests, as opposed to people working from the code to write tests to prove conformance with the spec, could apply. It would be good to see those sorts of success criteria baked into charters when the charter authors can't expect multiple implementations.
This thread isn't about the TTWG Charter specifically, though it does provide a useful example at the moment. Comments about that are welcome of course, in the appropriate place (i.e. the currently ongoing AC review).
Nevertheless, if as a result of this thread we end up specifying minimum license requirements for open source implementations to mitigate single implementation risks, those could and should be factored into WG charters, or better yet, they should be specified in the Process.
By the way, I think it's fine for a spec. to hang at CR waiting. The IETF has most of its work hanging at RFC…
@dwsinger I don't agree with this for two main reasons; I say this from individual experience, and am not representing the views of any organisation.
- It dilutes the value of W3C too much. Reaching Rec is an important signifier to other standards bodies, and it's important that the W3C version of the web does not become a closed shop, unreferenced externally. The norm for W3C is not the same as the norm for IETF, and relaxing it now would represent a change: that change is a signal to external viewers. (I do not consider it relevant that some standards bodies are already relaxed about the specifications they are willing to normatively reference from other places than W3C)
- Since we build specs on specs, it means we would be extending the already slightly shaky practice of normatively referencing CRs that are CRs because they don't meet their CR exit criteria, and therefore have increased risk to the community.
If you're arguing that the increased risk associated with CRs is in fact acceptable to the community, then why not relax the CR exit requirements and permit single implementations, for example?
It would be good to see those sorts of success criteria baked into charters when the charter authors can't expect multiple implementations.
I see two separate issues:
-
because there are multiple means of demonstrating implementation experience, it is not always wise/possible to bake the exact means of demonstrating implementation experience in a charter, especially when a WG has several deliverables. The W3C should therefore not require that charters specify the exact means of achieving implementation experience -- beyond those requirements in the process document.
-
I can see benefits in updating/expanding the list of means for demonstrating implementation experience in the process document to reflect current practices.
@nigelmegitt The risk is in referencing a document that has not been tested by interoperable implementation. Giving it the wrong label exacerbates risk, rather than reducing it. If you are referencing a CR, that makes it clear that there's a risk in your reference.
@dwsinger then the choice you are giving external organisations is that they can take the risk on themselves, and sometimes that results in unwanted additional cost, or broken functionality that impacts users, or they can avoid referencing W3C specifications and switch to some alternative.
I'm finding it very hard to see any strategic scenario in which this does not result in a loss of currency for W3C specifications and a loss of reputation for W3C.
I'd appeal to you and anyone else who thinks this is an acceptable direction to consider our constituency to be significantly broader than members' implementations and W3C specifications that normatively reference other W3C specifications. Even if you want to maintain a narrow view of W3C-only, we should not forget that W3C specifications that normatively reference CRs represents considerable risk that is not quantifiable.
@nigelmegitt I cannot see what extra risk being honest with the label brings to external organizations. Can you clarify?
Implementations have never been required to be from members, by the way, if that's what you are referring to (I am not sure).
I see serious risk to W3C's reputation overall, and to specifications that we publish without properly ascertaining their ability to support independent interoperable implementation.
@dwsinger ah, I did not mean that the risk is in changing the label, rather the risk is in encouraging, by acceptance, specifications to linger in CR semi-permanently. This seemed to be your view in https://github.com/w3c/w3process/issues/522#issuecomment-1051034143
@nigelmegitt I don't want to encourage it; just not delude ourselves by changing the label when our criteria are not met. I would much prefer that specs not "enter the box until their exit is clear" i.e. not enter CR status without an expectation of demonstrating independent interoperability. But if a spec. doesn't have demonstrated independent interoperability:
- we can change the label and mislead the community that it does
- we can leave it at CR.
I prefer the latter; the former carries lots of risk, not only to the customers of this spec. but to the consortium as a whole.
@dwsinger great, we're in agreement about that. The only question then is what to do when a spec has been in CR for so long that forwards-facing CR exit, in the direction of Rec, no longer seems like a plausible outcome.
