w3process
w3process copied to clipboard
w3c
Substantive changes undefined for Charter and Process reviews
Transferred from https://www.w3.org/community/w3process/track/issues/176 State: Raised
we need text proposals, please. editors to check that links correctly identify where terms are defined
Perhaps this needs defining in the possible Terms and Definitions section?
I think this issue has actually several sub parts:
- "Substantive change" and "editorial change" are only defined for specs, not for charters
- The usage of the term "editorial change" is inconsistent, and some place contrast Substantive changes to "minor changes" instead of "editorial changes".
- Daniel objects to step 2 of of 7.1.2 which allows the Director to make Substantive changes to charters or specs without returning the proposal for additional work.
I have made a pull request to solve 1 and 2: https://github.com/w3c/w3process/pull/217. I expect we might quibble on the details, but probably not on the principle.
As for 3, I first thought that this was best addressed as part of #182, but actually, there is a much simpler solution, which I have proposed in a separate pull request (https://github.com/w3c/w3process/pull/218): simply delete step 2 of 7.1.2. Making any substantive change to a document after the AC has voted on it pretty much invalidates the vote, as there is not reliable way of knowing how people would have voted had that change been included originally. We can (and should) go into further details as part of #182, but for now, just removing this possibility seems a good idea.
update: I've completed the https://github.com/w3c/w3process/pull/217 PR to define editorial vs substantive for the process in addition to for charters.
The process is currently undefined, since it says the director MUST do such and such when making substantive changes, yet does not define what substantive changes are for anything other than specs.
If we don't need to define it because it's obvious, why do we define it for specs? We do because it isn't obvious, and is controversial.
- During the CG call, we've had disagreements about whether changes to the schedule were substantive or not. I have seen (eh, made) formal objections based on a schedule being too short. I could imagine, let's say on WCAG since it has external timing constraints, formal objections due to a schedule being too long. To me, that means a schedule change is not editorial.
- You've commented that you thought that adding a deliverable to a charter was not a substantive change, I disagree. SVG was proposed to be added as a deliverable of the CSSWG at one point. There were strong objections to that, and it was eventually withdrawn. That sort of things is clearly not in the same category of things as spelling mistakes and broken markup (which is what editorial changes are for specs) that should be OK for the director to modify in a charter, after a vote, without saying why.
- The addition to the charter of the CSSWG, after an AC vote, of a clause opening up new ways for the CSSWG to do incubation caused the group to deadlock at the first F2F following that addition, and members to raise formal objections.
So no, I don't think the out-of-process guidance is good enough. I'm happy to argue out the details of what should or shouldn't be substantive, and what the Director can and cannot do when substantive changes are made, but not with leaving undefined, when I have seen it fail already several times in just the groups I have been personally involved in (and I would be surprised if this bad luck was unique to me).
For the record:
- I do not think the Director should have the right to unilaterally change the normative text of a spec, or to add or remove features, after the AC has voted on that text, even though the process currently allows it as long as he gives a rationale.
- I do not think the Director should have the right to unilaterally change the scope or the work mode of a working group, after the AC has voted, even though the process currently allows it (and is unclear as to whether he even needs to give a rationale for doing so).
- I do not think the Director should have the right to unilaterally change any and all part of the process, after the AC has voted, even though the process currently allows it (and is unclear as to whether he even needs to give a rationale for doing so).
- I think fixing typos, broken links, and the like, is OK to do, event after an AC vote, without having to provide a rationale
- I think that we can discuss whether there is a middle ground, with a certain range of changes that can be made unilaterally as long as a rationale is given. I am not convinced that very many things need to be in there, but we can talk. I am strongly convinced there should be limits.
I agree that changes to scope or deliverables are the very essence of substantive. Dates are almost certainly so as well.
Is there room here for "if anyone thinks it's substantive, it's substantive"? That is, for things in the grey areas, the Director writes "I propose doing X and I think it's not substantive; if anyone disagrees, say so, and we'll put the revised text to formal review".
As an aside, I'd like to keep the text as short as possible, of course, and give us as much latitude as possible...
I'm starting to think that splitting this PR into two may not have been a good idea after all, as the question of what is or isn't substantive is not that detached from what the Director is or isn't allowed to change by himself.
So far, we have 2 categories of changes (editorial / substantive), and 3 categories of actions (fix unilaterally without saying so fix unilaterally with a rationale / sent back to work). For the sake of this discussion, I'll just call "minor substantive" those issue that the director can fix himself as long as he justifies it, and "major substantive" those that get returned for more work. We can bikeshed the terms later if necessary.
Here is the process as it is today:
| Editorial | Minor substantive | Major substantive | |
|---|---|---|---|
| The Director can change without telling anyone | The Director can change unilaterally, but must provide rationale | The Director must send back to work and to re-vote | |
| Specs | fixing typos, fixing broken markup, clarifying informative notes, fixing bugs in informative examples | Everything else | up to the Director |
| Charters | undefined | undefined | up to the Director |
| Process | undefined | undefined | up to the Director |
I think the ability of the director to change absolutely anything unilaterally is a problem, and for charters and the process, the fact that it is unclear what changes he even has to explain at all is also a problem.
For Charters and the Process it is maybe debatable, but for specs especially, I do not quite see why the director needs to be able to make any normative change unilaterally.
The fact that the Director is a reasonable person unlikely to abuse his powers does not justify giving him absolute power to change anything at all.
What I propose instead:
| Editorial | Minor substantive | Major substantive | |
|---|---|---|---|
| The Director can change without telling anyone | The Director can change unilaterally, but must provide rationale | The Director must send back to work and to re-vote | |
| Specs | fixing typos, fixing broken markup | clarifying informative notes, fixing bugs in informative examples | Everything else |
| Charters | fixing typos, fixing broken markup | clarifying informative notes / sections extending the duration of a charter (since the Director can already do that based on 5.2.5 anyway) |
Everything else |
| Process | fixing typos, fixing broken markup | clarifying informative notes / sections | Everything else |
I hope that the proposal to not leave things totally undefined, as well as the proposal to have 3 categories, one of which includes things the director must send to rework and re-vote if he wishes to change, should not be particularly controversial.
As for what we fit in each of the 3 categories, I think there's more room for discussion. My position is probably quite on the strict side, though I don't think it is unreasonable in the light of the past controversies.
Based on what @dwsinger suggested, I would also be open to putting more things in category 2 if anyone complaining after the fact (within a certain time period) automatically reclassifies them as category 3, sends the proposal back to work and requires a revote.
Charters should be able to say whether deliverables can be added, within the scope, by decision of the WG, Director, or only on renewal of the charter.
I would also put fixing inconsistencies in the category of "can do, must provide rationale and they can be challenged to force an AC review". (For that matter, in specs too)
I have proposed an update to the Guide to add more detail: PR https://github.com/w3c/Guide/pull/31 Issue https://github.com/w3c/Guide/issues/32
As has been discussed in the CG calls, the proposed change to Process would not appear to address the situation that prompted concern, and Team's process for handling charter reviews has added significant re-review requirements since that time.
I recommend that we continue to fix things in the Guide and improve our processing there, rather than committing un-tested changes to the formal Process document.
I don't think that is sufficient.
While I am totally happy to argue about the specific text, I think there's a fundamental bug in the process as it is.
It tries to define two categories of changes, and to say that for one of the categories (substantive changes), the director MUST do certain things. But by leaving the two categories undefined for charters and for the process, we leave it to the appreciation of the director which change fall in which category, then this isn't a MUST at all.
So no, I don't think doing this in guidelines is appropriate. Guidelines are appropriate to establish jurisprudence and consistency in the way the director and the team acts within the bounds of the things they can/must do. They are not appropriate to define what is or isn't in their power to do.
Can we go with something short, that establishes principle rather than detail? "The following are always substantive changes to a charter:
- expansion of scope;
- addition of deliverable (which must, nonetheless, be in scope)
- extending a deliverable date to beyond the current charter period
- changes to the decision policy of the group The following are normally not substantive:
- editorial changes, improving wording, correcting spelling or grammar, etc.
- fixing broken links
- adding clarifying text."
I think that'd be an improvement over the currently undefined state, so I'd be in favor. We need something similar for the substantive change to the process, though.
And I still think that we also need to make it so that some of the changes need to be re-run by the AC. Currently all that is required for substantive changes is that the Director gives a rationale for what he changes, but he has the right to change anything. Without returning to a vote. I don't think that's right.
It seems we disagree strongly enough that we shouldn't aim this at Process 2019.
I still want to do this, but I don't see how we can get to agreement before the deadline for Process 2019, so removing Agenda+ for now.
As per https://www.w3.org/2020/01/29-w3process-minutes.html and https://www.w3.org/2020/02/12-w3process-minutes.html, this issue is deferred to a subsequent cycle of the Process.
A useful practical definition of a "substantive" change is a change - that in the judgement of the Director - would plausibly result in reviewers changing their response to a WBS poll (most notably if it would result in a formal objection). This may not have the rigor some are looking for.
Combining @jeffjaffe 's suggestion with my two above, yields a short list of things are normally substantive, a short list of things that normally are not, and the ability of someone in the team to make a judgment call on grey areas, which is announced (to the AC). If they think something is non-substantive, and anyone in the AC disagrees, we go with the more conservative view. Is that a lightweight enough process?
Putting it together:
In general, a "substantive" change is a change - that in the judgement of the Director - would plausibly result in reviewers changing their response during AC review.
The following, for example, are usually substantive changes to a charter:
- expansion of scope;
- addition of deliverable (which must, nonetheless, be in scope);
- extending a deliverable date to beyond the current charter period;
- changes to the decision policy of the group.
The following, for example, are usually not substantive:
- editorial changes, improving wording, correcting spelling or grammar, etc.;
- fixing broken links;
- adding clarifying text.
I would note that charters and WGs I've seen frequently add deliverables (in scope) and don't consider that a substantive enough change to have to recharter (as long as they're in scope). Deliverables also frequently slip past charter periods. :)
the problem is that the Director can make non-substantive changes without announcement or review, and if it's also the Director's judgment, people might not even notice when they would have disagreed.
The Revising W3C Process CG just discussed Substantive changes to charter.
The full IRC log of that discussion
<fantasai> Topic: Substantive changes to charter<wseltzer> ... ready to go?
<fantasai> github: https://github.com/w3c/w3process/issues/28
<wseltzer> florian: still have some concern
<cwilso> q+
<jeff> q+
<cwilso> q-
<wseltzer> ... when a charter is substantively changed director must seek re-review
<plh> rrsagent, generate minutes v2
<RRSAgent> I have made the request to generate https://www.w3.org/2020/08/26-w3process-minutes.html plh
<wseltzer> jeff: I said essentially, it's a judgment call
<wseltzer> ... if we don't want to leave it there, requires an enumeratioin of all cases, which is quite difficult
<wseltzer> ... dsinger_'s approach enumerated examples, but not claiming to be a comprehensive list
<dsinger_> q?
<fantasai> +!
<dsinger_> ack jef
<fantasai> +1
<wseltzer> florian: if the Director can make changes without telling anyone, we need to know what those cases are
<wseltzer> ... maybe in all cases, Dir must announce
<jrosewell> q+
<wseltzer> ... and then it's okay to have more leeway
<wseltzer> ... Dir must document rationale
<dsinger_> q?
<wseltzer> https://www.w3.org/Guide/process/charter.html#managing-changes
<dsinger_> ack jrose
<wseltzer> ^ Guide on managing changes to charters
<wseltzer> jrosewell: should we focus on processes for Director-free future instead?
<weiler> jrosewell++ for optimism
<wseltzer> dsinger_: we're trying not to introduce new Diretor instances
<dsinger_> q?
<wseltzer> jrosewell: we could park things as not making it worse
<wseltzer> florian: this is still open in Director-free
<wseltzer> ... Formal Objections go to the Council
<wseltzer> ... we haven't yet resolved whether they can only say yes/no, or make changes
<wseltzer> ... we could pund
<wseltzer> s/pund/punt/
In the outcome of an AC review, the director can make editorial changes to the document under review without announcing them, and it is also the Director's judgment whether a change is editorial. Note that he can make substantive changes as well, he just has to announce and explain them.
I think this can be solved in two different ways:
- as initially suggested here, reliably define what is or isn't substantive. For this purpose at least, since the document would be changed after having been reviewed, we probably want a fairly restrictive set of things to be changeable without saying so.
- require that editorial changes made to a document at the ends of its AC review must also be announced.
Given that writing a precise definition of what editorial means is hard, maybe we should go with 2 instead, that seems much easier.
This would mean: the following addition in 7.1.2:
This W3C decision is generally one of the following:
- The proposal is approved, possibly with editorial changes integrated. Any such change must be mentioned in the announcement.
I do realize that this is likely to be revisited once we go director-free, but the above would be a simple fix to a long standing issue.
The second level of the problem is that even though they require documentation of the change and of its rationale, the director is allowed to make unilateral substantive changes to a document after it's been voted on. I think that too is bad, but this is definitely something that will be revisited in the director-free process, so I'm OK with not addressing it here.
As to @cwilso's point in https://github.com/w3c/w3process/issues/28#issuecomment-680928860, 5.2.3 already lists certain substantive changes to a charter as being allowed without an AC Review, explicitly including the addition of in-scope deliverables. I think the way it does it is editorially a bit clumsy, and I'm thinking of proposing a clean up there as well, but I don't think there's a fundamental problem there.
See also: this comment https://github.com/w3c/w3process/issues/532#issuecomment-842940661
I believe this issue is addressed by the combination of:
- https://github.com/w3c/w3process/pull/647
- https://github.com/w3c/w3process/pull/646
I believe the examples in @dwsinger’s comment https://github.com/w3c/w3process/issues/28#issuecomment-679404039 should go into /Guide.
