OBOFoundry.github.io
OBOFoundry.github.io copied to clipboard
OBOFoundry
Governance: admit ontologies based on consensus or majority voting?
In rare cases, no consensus can be reached for admitting an ontology to the OBO foundry. This often relates to the question of whether specific logical modelling issues need to be addressed before the ontology can be admitted or not (not the issue here - please comment elsewhere).
The question for governance docs is: Should admission be based on consensus, i.e. a single OFOC member can block and admission, or on 2/3, simple majority voting? In the case of voting, should the vote happen transparently on the issue tracker item of the submission?
I think it would be clearer to say "unanimous consensus" rather than just "consensus".
Also should clarify who votes and how it's counted. Like, is the alternative to the vote happening on the issue tracker that the vote done by whoever is on the OBO Ops Committee call where the new ontology is discussed?
If the vote is limited to the OFOC, then it's a small group of people who understand the issues well. In that case I would expect objections to be well founded. Seems to me unanimous consensus is appropriate.
The problem is the converse: unanimous consensus means a single person can decide to reject an ontology submission request.. I am not saying anything for or against either option just taking a note. I think the expert argument is hard here because 1 expert against means 9 10 experts in favour..
In the meeting we keep being held up about ontologies doing some bad modelling choices, with the OBO committee being split in the question of wether the bad modelling is enough to hold off submission until fixed, or wether we can accept trusting that the submitter will address the modelling issues. Principle wise, bad modelling is rejected under "scientific accuracy". My problem is that on the one hand I doubt there are any ontologies without modelling mistakes (aside from BFO maybe :P) and quantifying the "degree of badness" is subjective; on the other hand some people on the committee maintain that scientific accuracy principle is a central tenant of OBO (so central it's in our mission statement) so are inclined to block whenever they see a wrong existential restriction. That is IMO why we should consider majority voting. Even absolute majority like 66+%. A rule like "bad existential restrictions are allowed" will never be passed to operationalise this problem.
Yes, I understand. I guess I'm thinking that all of the people on OFOC are sufficiently capable and responsible that even if one blocks, there's good reason not to go forward. FWIW, in my experience the likelihood that sufficient modeling changes will be done post admittance is low. What is the downside to asking that the changes be made before being admitted?
The vagueness of it.. we either review all axioms or no axioms, but it does not make sense to randomly pick 16 existential restrictions from 2000 say why they are wrong and then say fix them before admittance. Either we say: all axioms must be correct, and then we have to review every single one, or we don't make correctness of individual axioms a formal acceptance criterion.
Or, you tell them which axioms are problematic and why, and ask them to fix all that have these problems. Then, on resubmission, you review another random sample, repeating until the ontology meets a level that everyone is comfortable with, or until everyone has confidence that the rest of the necessary changes will indeed be made. Seeing how they respond to the first round of requested changes will give some priors on whether or not to believe that necessary changes will be made in the future.
An then there's the question of whether this policy on consensus would be decided by consensus or majority :-)
Didn't you advice a more lightweight process once? I think this will just lead to super inconsistent reviews; and months long submission processes.
Do you think there's any process that will lead to consistent reviews? Won't a process that iterates random samples get you closer to consistent than a process that does a single random sample?
As far as time spent, I haven't been working on OFOC in a while, so I don't know the pace. How long are you currently spending on identifying problems in the ontology? How much time are you spending trying to get to consensus? The suggestion I made assumed that too much time was being spent on getting to consensus. I'm imagining that if a consensus check can be done quickly, and when it fails it's clear what to do next, then overall more time might be spent on quality and less in stressful discussions. But I may have guessed the situation wrong.
I'm not sure what I said about a lighter process. Lighter than we tried early on, since that didn't work. Maybe what you are doing now is unworkable too? Based on the discussion I also assumed that there's agreement that some level of manual review is worthwhile. My suggestion was based on that assumption.
I added my thoughts to #1861 which is where this discussion should be continued. The issue here is only about gathering opinions on the voting to prepare a vote on the voting.
Suggest to follow the transparent PlosOne publish model. Whoever approved or dis-approve, can put their comment online and make it available for everyone to view.
Are there still action items in this issue, or can we close since @cstoeckert opened #1861?