OBOFoundry.github.io
OBOFoundry.github.io copied to clipboard
OBOFoundry
Remove foundry distinction on front page
We should remove the priority ordering of foundry ontologies on the front page, as discussed on numerous calls.
- no new reviews have been completed since 2016. Prior to the one review in 2016, the last review was 2013
- this makes us look very bad
- it confuses the community. From @mellybelly: I continue to hear that people come to the OBO page and disregard the ontologies that are not "Foundry"; e.g. the OBO Foundry does not have many production ontologies."
- while the process seems to be incrementally improving at the current rate we are not on track for making meaningful progress in the near future
- we will likely soon have a much better process where we provide more meaningful metadata e.g. dashboard but in the interim we should just drop the misleading distinction
EWG note: There have indeed been reviews since 2016, publicly available from the Foundry website (http://www.obofoundry.org/docs/CompletedReviews.html), though it seems the newest are not shown.
Can we encourage more ontology developers to submit their ontologies for review ? And show which ontologies have been reviewed with respect to the principles. And also show the dashboard results for the ontologies. Lynn
The fact remains that there are many great ontologies in the OBO library and yet we promote only those that have had OBO editorial review. Many ontologies within the library are mature, in wide use, and have had peer review in numerous other contexts. The foundry ordering/designation misrepresents and diminishes our community imho.
Would be great to have those mature, widely used ontologies reviewed. Any reason not to submit them for review ?
Submit for review: The Foundry listing is just show which ontologies were reviewed and comply with the Foundry principles. Since the principles are core to the Foundry, why wouldn't we want to show which ontologies comply to the principles.
The Dashboard will also be a great resource to show how ontologies comply to the principles.
Strictly speaking, Foundry designation has one real purpose: to indicate which of the (possibly several) ontologies in a given domain should be the point of 'convergence' (that is, reference and reuse by other ontologies). That's it. The manual reviews were meant as a way of selecting these. That's it. Unfortunately, every time we discuss abolishing the distinction between Foundry and Library, the discussion centers around reviews and how many were done and how the process works, and misses the main point. Given the developments around streamlining and automating the review process, I think we need to return to the central question: Do we find any value in selecting specific ontologies as points of convergence? If the answer is "No" then by all means get rid of the distinction completely. If the answer is "Yes" then consider the following proposal:
Decouple Foundry designation from manual review status, except in certain cases (see below). The designation can be given right away to any ontology that covers a completely unique sub-domain (that is, one that doesn't have overlap with another ontology). We would need a consensus on which ontologies meet this criterion. Ontologies that do have overlap with one or more other ontologies will need to be manually selected to get the Foundry designation, and that selection process can include manual reviews. We can decide that such reviews must be done in tandem (that is, the two or more conflicting ontologies get reviewed at once), or we can decide that whichever ontology makes the request gets reviewed, with favorability points given for the decision to be engaged in the process. Any unreviewed ontology with the Foundry designation would have that designation revoked if another ontology within the same sub-domain appears.
The advantage of this approach is that it encourages some of the very things we want to encourage, such as development of non-overlapping ontologies, yet maintains the purpose of the Foundry distinction. I imagine there would be quite a lot of ontologies with this designation using this approach, thus minimizing the issue of "it looks bad not having so many ontologies with Foundry status". Manual reviews can still be performed as insurance against revoking Foundry status.
Great idea Darren, +1 for your proposal
Cheers, Lynn
Sent from my iPhone
On Apr 14, 2020, at 1:43 PM, Darren A. Natale [email protected] wrote:
Strictly speaking, Foundry designation has one real purpose: to indicate which of the (possibly several) ontologies in a given domain should be the point of 'convergence' (that is, reference and reuse by other ontologies). That's it. The manual reviews were meant as a way of selecting these. That's it. Unfortunately, every time we discuss abolishing the distinction between Foundry and Library, the discussion centers around reviews and how many were done and how the process works, and misses the main point. Given the developments around streamlining and automating the review process, I think we need to return to the central question: Do we find any value in selecting specific ontologies as points of convergence? If the answer is "No" then by all means get rid of the distinction completely. If the answer is "Yes" then consider the following proposal:
Decouple Foundry designation from manual review status, except in certain cases (see below). The designation can be given right away to any ontology that covers a completely unique sub-domain (that is, one that doesn't have overlap with another ontology). We would need a consensus on which ontologies meet this criterion. Ontologies that do have overlap with one or more other ontologies will need to be manually selected to get the Foundry designation, and that selection process can include manual reviews. We can decide that such reviews must be done in tandem (that is, the two or more conflicting ontologies get reviewed at once), or we can decide that whichever ontology makes the request gets reviewed, with favorability points given for the decision to be engaged in the process. Any unreviewed ontology with the Foundry designation would have that designation revoked if another ontology within the same sub-domain appears.
The advantage of this approach is that it encourages some of the very things we want to encourage, such as development of non-overlapping ontologies, yet maintains the purpose of the Foundry distinction. I imagine there would be quite a lot of ontologies with this designation using this approach, thus minimizing the issue of "it looks bad not having so many ontologies with Foundry status". Manual reviews can still be performed as insurance against revoking Foundry status.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
It is not clear that without improved OBO community governance that there is convergence on what function the reviews serve. Perhaps a survey would be useful to try to understand this.
I do agree that there should be a determination of whether a given ontology meets (and initially, to agree to) the OBO principles. The great work that the technical team is doing will be a much more efficient and largely automated process to demonstrate this.
Would be great to have those mature, widely used ontologies reviewed. Any reason not to submit them for review ?
Numbers of ontologies and speed of review means we're unlikely to get through them? Given that most of the checks can now be automated, why not ditch the manual step and just have a small set of sortable rankings based on the dashboard? There are lots of interesting and creative ways we could extend dashboard tests: logical consistency; % inferred classification; integration scores - import from other ontologies & use by other ontologies. Why not focus efforts on this?
Strictly speaking, Foundry designation has one real purpose: to indicate which of the (possibly several) ontologies in a given domain should be the point of 'convergence' (that is, reference and reuse by other ontologies)
It doesn't look that way to the outside world - which I think is what counts here. It also doesn't come near to reflecting the number of non overlapping or minimally overlapping ontologies in the library. Ignoring the question of how this decision is reached or reviewed for now, wouldn't this be better communicated with a clear, designated status that is one more column in a sortable table?
My proposal addresses how it looks to the outside world AND maintains one of the key reasons why the Foundry exists: "By joining the initiative, the authors of an ontology commit to working with other members to ensure that, for any particular domain, there is convergence on a single ontology." (quote is from the 2007 Foundry paper). @dosumis that proposal is not mutually exclusive with the idea of a new sortable column (which really doesn't help the "how it looks" issue, it just potentially hides it until the sort is done). Your statement that "it doesn't come near to reflecting the number..." I presume refers to the situation now? Because unless you think there are very few cases that will pass my proposal super-minimal criteria, it absolutely WILL come near to reflecting the ontologies. I imagine not many will need comparison.
I am a bit new to all this, and I wouldn't take my opinion too serious because I really don't know what the long term implications are for encouraging convergence vs. not. I certainly see no problem with a flag "designated point of convergence", which can be used for filtering and sorting as David suggests, but I think retrospectively going through 150 partially overlapping ontologies and debating them will not be worth the cost. I would prefer we focus our limited resources on alignment with OBO Core, and invest a bit more on standardising automated alignment strategies between overlapping ontologies. I think all Foundry ontologies are gifts to the world, of different levels of maturity, for all sorts of applications, and there is no need that the obofoundry website highlights ontologies that at some point past underwent some review process, which we don't have the resources to repeat regularly enough (imagine "green" ontologies going stale - who is going to keep track). I am generally unsure about the value of convergence (really unsure, in both directions), so I won't cast a vote wether or not we should continue to do an editorial review process to that end (I would love to debate it though); but I think the highlighting on the overview page should go, if necessary in favour of a sortable column as @dosumis suggests.
@nataled last comment almost made me retract my statement because of the reference to our 2007 paper, which seems fair enough (building on the founding principles is a good thing) - right now I am just worried that we are underestimating the effort of determining "non-overlapped-ness", given the other things we could and should be doing.
@matentzn I wasn't thinking that we'd do a full-scale analysis on a term-by-term basis or anything like that. More like just review the list and find those that are uncontroversial. I'd be surprised if finding such a seed list would take more than a day. If we wanted to be a bit more methodical about it, I'd say we have 3 or 5 volunteers go through the list and make their selections, individually, and we take those that have 100% agreement. Those that have less than 100% agreement get discussed at an Ops meeting, and those that weren't suggested for inclusion by any of the volunteers get a deeper look, flagging some for possible review.
The more I think about it, the better I like it. We maintain everything the Foundry stands for, we get the sortable column (which I think is for 'went through manual review'), and we get to focus our attention on those cases that need it (for example, right now we are reviewing ontologies that would very easily pass the no-conflicts criterion).
@dosumis:
Given that most of the checks can now be automated, why not ditch the manual step and just have a small set of sortable rankings based on the dashboard?
Yes please!!!!
'Strictly speaking, Foundry designation has one real purpose: to indicate which of the (possibly several) ontologies in a given domain should be the point of 'convergence' (that is, reference and reuse by other ontologies) It doesn't look that way to the outside world - which I think is what counts here.
Exactly!!!
From meeting on 5/19: Many were okay with changing ordering to simple alphabetical, keeping stars, but there were some who felt it was worse than the current configuration. Many options for improving the list were suggested and will be discussed on separate issues. Since we could not reach a consensus on this issue, a simple vote will be held to choose between keeping as is or simple alphabetical ordering with stars.
I have volunteered to organise a vote, but I ask kindly to make one stab at a proposal that is (1) feasible as a quick temporary solution and (2) I am confident could get consensus (rather than having two camps from which one will be upset). I will describe the solution at the next meeting.
I had previously opened a ticket (#1217) to discuss the proposal I made earlier, but it is clear from the discussion there that such proposals won't be given their due until we come to a firm agreement (or majority vote, if it comes to that) on whether or not Foundry distinction is desired. I closed that ticket, but please do read the comments there, as they are about the issue being discussed here.
I'd like to see the stars become "reviews available" and to make those reviews living documents. The Zebrafish Anatomy ontology was submitted by myself more than 10 years ago, I think. The reviews today would be very different than long ago. It would be better if we had a public guideline and/or template for submitting reviewer comments that could be kept up by the community. I think this would be more current, more democratic, and inspire more discussion about the issues found. We would learn from eachother more. And it would be complementary to the automated checks. We could support a more innovated, open review process - it is what the "O" means ;-).
It looks to me like a star system doesn't get around the controversies - in the version proposed in #1217 it privileges manual review just as the current page does. I favour a simple table with sortable columns, some of which provide a condensed report of automated review status and with at least two columns referring to manual manual review: one with review status (pass/fail/unreviewed) + one or more additional columns providing details (review date; link to review).
While I agree that convergence is desirable, it is also, clearly, hard to achieve. Convergence is desirable, at least in part, because we want a unified, well integrated set of ontologies. I'd be very interested to investigate (semi?)-automated scoring systems to measure this, e.g. re-use of terms from other foundry ontologies & relations from RO in a manner that does not fail constraints; curated equivalence mappings/bridge axioms.
18 months and still no change on the front page, still no reviews. Uberon was requested almost 2 years ago, I requested all material be made public but nothing has been made available #1102
we should prioritize a sortable table as suggested here
This is very easy to do as static html, e.g see @cthoyt's page https://cthoyt.com/obo-community-health/
This also has more meaningful metrics than whether an ontology was reviewed 10 years ago
A question was asked in Slack:
so a minor branding question. Back in the day (about 5 years ago when I tried to understand) "OBO Foundry ontologies" meant "Ontologies that met all the OBO Foundry criteria" (about 10), and all the rest were called "OBO Library ontologies". BioPortal has an OBO Foundry checkbox that shows the principal 10 or so ontologies. So in BioPortal we can either show all the ontologies from OBO Library as OBO Foundry ontologies, or add a checkbox called OBO LIbrary for showing all of them. Preference?
Thank you for sharing this post Nomi. This topic has been discussed for some time, without resolution.
Could we devote one of our upcoming OBO Operations calls to this topic ?
Cheers, Lynn
On Thu, Jan 6, 2022 at 5:07 PM Nomi Harris @.***> wrote:
A question was asked in Slack:
so a minor branding question. Back in the day (about 5 years ago when I tried to understand) "OBO Foundry ontologies" meant "Ontologies that met all the OBO Foundry criteria" (about 10), and all the rest were called "OBO Library ontologies". BioPortal has an OBO Foundry checkbox that shows the principal 10 or so ontologies. So in BioPortal we can either show all the ontologies from OBO Library as OBO Foundry ontologies, or add a checkbox called OBO LIbrary for showing all of them. Preference?
— Reply to this email directly, view it on GitHub https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1140#issuecomment-1006969697, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBB4DMZR42PPCFEWNPCY2LUUYHDRANCNFSM4K5XWQEA . You are receiving this because you commented.Message ID: @.***>
-- Lynn M. Schriml, Ph.D. Associate Professor
Institute for Genome Sciences University of Maryland School of Medicine Department of Epidemiology and Public Health 670 W. Baltimore St., HSFIII, Room 3061 Baltimore, MD 21201 P: 410-706-6776 | F: 410-706-6756 @.***
Great !! Thank you Nomi !!
On Fri, Jan 7, 2022 at 4:47 PM Nomi Harris @.***> wrote:
Yes; I'd already added it to the agenda for the next call on Jan 25.
— Reply to this email directly, view it on GitHub https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1140#issuecomment-1007763126, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBB4DIVCQBZIAB7QRRQEOTUU5NPPANCNFSM4K5XWQEA . You are receiving this because you commented.Message ID: @.***>
-- Lynn M. Schriml, Ph.D. Associate Professor
Institute for Genome Sciences University of Maryland School of Medicine Department of Epidemiology and Public Health 670 W. Baltimore St., HSFIII, Room 3061 Baltimore, MD 21201 P: 410-706-6776 | F: 410-706-6756 @.***
As Lynn correctly pointed out, we have been discussing this over and over, and there is a lot of history with these labels that evoke strong reactions, and I doubt that a discussion on the 25th would resolve this.
I am wondering if we can benefit from a complete change of labels, with less baggage. We could identify what the 'states' are that we are trying to distinguish. Give them labels that are actually understandable. And at the same time preserve the history and effort put in by everyone.
state #1: 'OBO member ontology'. - an ontology listed on the OBO registry, with an OBO namespace. These come with attributes indicating their current development (active / inactive / orphaned / obsoleted) state #2: 'OBO project ontology' - an OBO member ontology designed for a specific project as its primary use case. These ontologies are not designed for perfect interoperability with others, and may not adhere to all community requests state #3: 'OBO domain ontology' - an OBO member ontology designed to capture a defined scientific domain. These ontologies are designed to capture terms within a domain for all users requesting them and will work on resolving issues to make that happen
I hope until here it is uncontroversial. #1 is well defined. #2 would be claimed by the ontology developer who wants to signal that they are not open to re-arranging their ontology hierarchy because of e.g. 'axiom injection' issues that affect others but not their project. #3 is a goal to have that ontology developers have to put work in.
Then comes the controversial part:
state #4: 'OBO preferred domain ontology' - an OBO domain ontology that has been designated as the preferred source of terms within a domain.
The separation of #3 and #4 is where the real controversy has been, and #4 is where the 'OBO foundry' designation comes from. But our 'branding' is confusing, as the website is called obofoundry.org, we constantly confuse OBO / OBO library / OBO foundry etc.
Let me know what you think.
Bjoern
On Fri, Jan 7, 2022 at 1:50 PM lschriml @.***> wrote:
Great !! Thank you Nomi !!
On Fri, Jan 7, 2022 at 4:47 PM Nomi Harris @.***> wrote:
Yes; I'd already added it to the agenda for the next call on Jan 25.
— Reply to this email directly, view it on GitHub < https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1140#issuecomment-1007763126 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABBB4DIVCQBZIAB7QRRQEOTUU5NPPANCNFSM4K5XWQEA
. You are receiving this because you commented.Message ID: @.***>
-- Lynn M. Schriml, Ph.D. Associate Professor
Institute for Genome Sciences University of Maryland School of Medicine Department of Epidemiology and Public Health 670 W. Baltimore St., HSFIII, Room 3061 Baltimore, MD 21201 P: 410-706-6776 | F: 410-706-6756 @.***
— Reply to this email directly, view it on GitHub https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1140#issuecomment-1007764628, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADJX2IX252GO57RGKGH4WKTUU5NZVANCNFSM4K5XWQEA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
-- Bjoern Peters Professor La Jolla Institute for Immunology 9420 Athena Circle La Jolla, CA 92037, USA Tel: 858/752-6914 Fax: 858/752-6987 http://www.liai.org/pages/faculty-peters
I think this is a great step forward, I think this is exactly the right spirit. Categorizing in this way will be a real help to our users.
I might prefer a different label for "2" but that's a minor issue
If 4 is controversial, then we can still make incremental progress by starting with 1-3.
And we can simply inform our users if there are multiple ontologies for any given domain. This could be automated. For example, on the page for the sheep ontology it might have some neutral autogenerated text:
"note there is another ontology with overlapping content, Bjoern's sheep ontology[link], at this time OBO does not favor one over the other, consult documentation on both ontologies to decide which best fits your use case"
I think the challenge is how to we decide what the domains are (and do this defensively to avoid gaming)
COB is a sensible choice, but I think it has to be more granular, such that we can eventually aim to have a single preferred domain anatomy or phenotype ontology at any given taxonomic level. But we have to have some principled shoreline, otherwise people will try and game things to turn their no2 ontologies into no3 ontologies ("of course my ontology of sheep wool patterns of Northern Wales is a domain ontology").
Can an ontology claim more than one domain?
I think this would be useful. Consider behavior. There is NBO whose scope completely matches this domain. But it is functionally inactive (the only commits in the last 2 years are technical, not content). Then there is GO, which has some behavior terms, some of which it does a good job of, sometimes not so much as it's not our area. The reasons for this impasse are complex, but it is how it is, and I think we do our users a service by dispassionately stating that the scope of NBO is behavior, and the scope of GO overlaps behavior.
Of course if COB forms our domains then we already have some of this information in the form of the cob-to-external axioms.
Message ID: @.***
com>
Nice yeah, fully behind @bpeters42 and @cmungall last two comments. I would also recommend to tackle 1-3, and drop 4 for now, and use the domain tagging mechanism to group related ontologies. AFAIK there is only a handful of controversies, and this should not hold us up. I am also fine with multiple domains, but I saw the current domain mechanism more as a way to group related ontologies into blocks for display, not so much a fully account of all kinds of branches in an ontology. I think that discussion could also be head in COB, and here we agree on a few rough "main" categories for display.
Discussed during OBO Operations meeting on 25-Jan-2022. The set of "OBO member ontologies" should be comprised completely of ontologies that are either project-specific ontologies or domain ontologies. Our next step is to look at the current set of OBO Foundry ontologies and try to classify them as one of those.
BTW see also Reference ontologies vs. "Other" kinds #1546
from @bpeters42 in obo ops email:
I think the person to fill these out first is the ontology maintainer, who is listed as the contact. That is what I meant by 'self-assessment'. We can have separate columns of comments of discussions how other disagree, but I would want to separate those from what the person who is working on the ontology thinks.
I agree
We should have some clear criteria that can help guide them in their self-assessment, and in our evaluation.
I think one of the strongest indicators will be imports. A domain ontology will likely be imported either directly or via robot extract / ontofox, and its URIs reused by another OBO ontology. A project specific ontology is less likely to be imported by others. We could likely do an ontobee query for this. This would not be hard and fast but it can help.
What I think the next steps are:
1, A PR that is both (a) an extension to the json schema to add
a. foundry_category, which will optional and an will be an enum of:
-
project_specific_ontology -
domain_ontology -
undecided
note there is information is explicitly selecting undecided vs leaving blank, we can use this for difficult or contested states
b. primary_domain
This will be mandatory IF domain_ontology is the category, otherwise it must be blank
In the interests of incremental progress, this can start as a string, with some guidance as to what will go there, and we can iterate on turning this into an enum. This is a separate ongoing piece of work here: #1225
2, merge the PR
3, email to obo-discuss asking community to make PRs on their ontologies to select category and fill in existing data
it is OK if some take time to fill these in. We expect domain ontologies to be more responsive and the priority is to get these assigned
4, make minimal changes to the liquid template such that domains listed first
if someone wants to take on the extra work of doing a faceted browser, go for it, but in the interests of moving forward let's start with this fixed ordering
Agree with all, but #3 - I would suggest we start off emailing the obo-operations group, asking them to do this for ontologies they are maintainers of. That should discover problems and needs for better documentation before emailing all at obo-discuss. Not meant to be exclusive with this, other ontologies / people should weigh in, but the obo-operations group at least should be up to date in the discussion, and I don't mind sending them multiple revised instructions as we iron potential problems out.