RTXteam
Investigate why we are failing a few tests that Aragorn passes
https://arax.ncats.io/devLM/index.html?systest=115 [filter for tests that ARAX fails, but Aragorn passes]
Probably each one of these will be a story unto itself, and I suggest creating an issue for each as someone starts down each rabbit hole, perhaps referencing this issue.
The latest test results show that ARAX and the test harness are now performing pretty well. ARAX is getting nearly 81% correct, which is nice. I think we have ironed out all the systemic problems: https://arax.ncats.io/?systest=116
I think it would still be a very useful exercise to start working through some of the tests that we FAIL, but either Aragorn or Unsecret (other high performing ARAs) do PASS, and try to understand why we fail and if there is a sensible course of action to PASS.
Probably each one of these will be a story unto itself, and I suggest creating an issue for each as someone starts down each rabbit hole, perhaps referencing this issue.
re melphalan - nemaline myopathy https://arax.ncats.io/beta/?r=313412 only direct ARAX connection is https://arax.ncats.io/beta/?r=313412 CHEBI:28876----biolink:treats_or_applied_or_studied_to_treat----MONDO:0018958 Id: infores:automat-robokop:CHEBI:28876--biolink:treats_or_applied_or_studied_to_treat--None--None--None--MONDO:0018958--infores:text-mining-provider-targeted
So if "treats_or_applied_or_studied_to_treat" is a synonym for "treats" then we actually pass this one. Maybe we need to have the testing algorithm recognize "treats_or_applied_or_studied_to_treat" and "treats" as equally acceptable.
Perhaps not surprisingly, we get exactly the same answer directly from the ROBOKOP interface https://robokop.renci.org/question-builder/answer "treats_or_applied_or_studied_to_treat" So it is quite odd that ARAGORN passes, and ARAX does not.
Unless perhaps an older version of ARAX is being tested??
fwiw, the two-hop answers are here https://arax.ncats.io/beta/?r=313414 interesting, but no real additional insight from these.
I (re)notice that there are a substantial number of instances of NeverShow tests that Aragorn passes but ARAX fails based on data it is getting from automat-robokop. On a whim, I filed an issue in the Tests repo: https://github.com/NCATSTranslator/Tests/issues/116
Hoping to get a response from Aragorn/Robokop folks
A good test query to see this problem is:
{
"edges": {
"t_edge": {
"knowledge_type": "lookup",
"object": "ON",
"predicates": [
"biolink:affects"
],
"qualifier_constraints": [
{
"qualifier_set": [
{
"qualifier_type_id": "biolink:object_aspect_qualifier",
"qualifier_value": "activity_or_abundance"
},
{
"qualifier_type_id": "biolink:object_direction_qualifier",
"qualifier_value": "increased"
}
]
}
],
"subject": "SN"
}
},
"nodes": {
"ON": {
"categories": [
"biolink:Gene",
"biolink:Protein"
]
},
"SN": {
"ids": [
"CHEBI:29103"
]
}
}
}
A covert ops team has temporarily blocked querying to infores:automat-robokop to see how this affects performance.
Last Friday, additional code was inserted to block querying of automat-robokop completely. Committed into master and thus CI and also arax.ncats.io/test, /beta, /devED. (not main)
The result is a boost in ARAX performance on the weekend automated tests https://arax.ncats.io/?systest=183
Passing 17 more (NeverShows), but failing 3 more for a net gain of 14.
The great majority of the new passing ones are Test Case 4 (worth noting that these tests used to pass several months ago) :
The newly-failing tests also show a reversion:
[ n.b. Run 156 is from early May 2025 ]
Next step: what to do?
