upgrade version python used for ARAX to python3.12

[saramsey]

I know this was brought up at an ARAX AHM recently, but I am resurfacing the matter as an issue.

Hi all, we're currently using python3.9 in ARAX's flask application, and that is a pretty old version of cpython. Since 3.9, the PEP484 type hinting system has been vastly improved in terms of ergnonomics and expressiveness, enabling much more precise pre-runtime analysis of python code, to catch bugs before testing inside flask and before unit testing. So for that reason alone, I think the system could benefit by transitioning to a new version of python. If we can bump all the way to 3.12, we might get a speed improvement, as a side bonus.

[edeutsch]

sure, this would be fine by me. We'd have to hope that our tangle of dependencies would be good on Python 3.12.

Is 3.12 everyone else's vote as well? Any reason to select a different version?

[saramsey]

3.12 is fine with me

[bazarkua]

I’ve investigated RTX/code/UI/OpenAPI/python-flask-server and confirmed it imports ARAX modules (e.g. RTXConfiguration, ARAX_database_manager, ARAX_background_tasker). This means Flask app must run in the same Python interpreter. And since ARAX is pinned to Python 3.9, this forces the Flask process to run under 3.9. Because of that we can't upgrade Python version just for ARAX Flask app. __main__.py:

`sys.path.append(os.path.dirname(os.path.abspath(__file__)) +
                "/../../../../ARAX/ARAXQuery")
sys.path.append(os.path.dirname(os.path.abspath(__file__)) +
                "/../../../..")

from RTXConfiguration import RTXConfiguration
from ARAX_database_manager import ARAXDatabaseManager

I think there are two possible ways:

1. We would need to upgrade full ARAX codebase to run under Python 3.12 (a lot of dependencies would need to be upgraded, and changes to dockerfile would need to be made as well)
2. Another solution which I'm not sure about is to isolate the Flask app via running it on it's own docker container to keep ARAX core on 3.9, we would need to some workarounds so Flask app doesn't use modules from ARAX directly.

[saramsey]

We have a bunch of dependabot warnings about dependencies that need to be updated anyhow. I lean toward trying to upgrade everything to run in Python 3.12.

Since the full ARAX already runs inside a Docker container, trying to change ARAX so that only the Flask app runs in a Docker container seems like an unappealing option. I'm not saying it couldn't be done. But rather, I think trying to get the full ARAX working on Python 3.12 sounds like the more.... appealing of the two options, by comparison.

Just my off-the-cuff $0.02.

[edeutsch]

I never considered option 2 as a viable option, and assumed we were discussing option 1.

[bazarkua]

Migrated ARAX from Python 3.9 to Python 3.12 version code in issue2521: https://github.com/RTXteam/RTX/tree/issue2521

Updated requirements.txt with Python 3.12 compatible versions

• Upgraded numpy (1.26.4), pandas (2.2.1), scipy (1.13.1), scikit-learn (1.4.2)
• Updated Flask (3.0.2), SQLAlchemy (2.0.28), neo4j-driver (5.18.0)
• Removed asyncio (now built-in to Python 3.12)

Code Fixes:

• Fixed Flask JSONEncoder import for Flask 2.3 compatibility
• Resolved pkgutil.ImpImporter removal by updating dependencies
• Handled distutils removal using ensurepip
• Updated Docker containers to install Python 3.12

Run pytest suite 143 tests pass

Also tested Flask app startup and functionality Next I will try to test issue2521 branch on arax.rtx.ai or arax.ncats.io/beta

[bazarkua]

@chunyuma After testing example 3 query on arax.rtx.ai as an effort to verify successful migration from python3.9 to python3.12, I've encountered this error:

2025-07-25T06:39:21.212167 INFO: (57283) [] loading embeddings and models into memory
/home/ubuntu/venv312/lib/python3.12/site-packages/sklearn/base.py:442: InconsistentVersionWarning: Trying to unpickle estimator DecisionTreeClassifier from version 1.0.2 when using version 1.7.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
2025-07-25T06:39:27.692841 ERROR: (57283) [UncaughtARAXiError] An uncaught error occurred: node array from the pickle has an incompatible dtype:
- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]: ['Traceback (most recent call last):\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_query.py", line 791, in execute_processing_plan\n    expander.apply(response, action[\'parameters\'], mode=mode)\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_expander.py", line 328, in apply\n    response, overarching_kg = self.get_inferred_answers(inferred_qedge_keys, query_graph, response)\n                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_expander.py", line 722, in get_inferred_answers\n    infer_response = inferer.apply(response, infer_input_parameters)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_infer.py", line 366, in apply\n    getattr(self, \'_\' + self.__class__.__name__ + \'__\' + parameters[\'action\'])()  # thank you https://stackoverflow.com/questions/11649848/call-methods-by-string\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/UI/OpenAPI/python-flask-server/openapi_server/../../../../ARAX/ARAXQuery/ARAX_infer.py", line 619, in __chemical_gene_regulation_graph_expansion\n    XCRG = creativeCRG(self.response, os.path.sep.join([*pathlist[:(RTXindex + 1)], \'code\', \'ARAX\', \'ARAXQuery\', \'Infer\', \'data\', \'xCRG_data\']))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/ARAX/ARAXQuery/Infer/scripts/creativeCRG.py", line 216, in __init__\n    self.increase_model = load_ML_CRGmodel(self.response, self.data_path, \'increase\')\n                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/RTX/code/ARAX/ARAXQuery/Infer/scripts/creativeCRG.py", line 110, in load_ML_CRGmodel\n    fitModel = joblib.load(file_path)\n               ^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 749, in load\n    obj = _unpickle(\n          ^^^^^^^^^^\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 626, in _unpickle\n    obj = unpickler.load()\n          ^^^^^^^^^^^^^^^^\n', '  File "/usr/lib/python3.12/pickle.py", line 1256, in load\n    dispatch[key[0]](self)\n', '  File "/home/ubuntu/venv312/lib/python3.12/site-packages/joblib/numpy_pickle.py", line 446, in load_build\n    Unpickler.load_build(self)\n', '  File "/usr/lib/python3.12/pickle.py", line 1760, in load_build\n    setstate(state)\n', '  File "sklearn/tree/_tree.pyx", line 836, in sklearn.tree._tree.Tree.__setstate__\n', '  File "sklearn/tree/_tree.pyx", line 1542, in sklearn.tree._tree._check_node_ndarray\n', "ValueError: node array from the pickle has an incompatible dtype:\n- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}\n- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]\n"]

it's happening because XCRG models were created with scikit-learn 1.0.2 in a python 3.9 environment (old) and scikit-learn 1.0.2 is very old and not compatible with python3.12 the oldest we can use is scikit-learn==1.2.2 that's still compatible with python3.12 but it's causing this error, because the internal data structures changed in newer versions of scikit-learn

[chunyuma]

Hi @bazarkua，thanks for bringing up this issue. It seems like this problem might not be resolved until we retrained the xCRG model with the updated python and scikit-learn version. It may need to take up to two weeks to make the retraining process done because the GraphSage model, which was used to train xCRG, has been updated for a while. I am not sure whether it can be compatible with python 3.12 and the updated scikit-learn package. If not, I need to find the alternative model. Is it urgent for this update? Is it ok to allow me to retrain the model in the next two w? @saramsey

[edeutsch]

I am wondering if it might be easier/reasonable simply to write a trivial script to unpickle the file(s) and write to JSON under Python 3.9. And then another trivial script to read the JSON and write out pickle files and run under Python 3.12?

[chunyuma]

Thanks for the great idea @edeutsch. I have just consult ChatGPT with the possibility of this idea. It seems like using the skips.io might allow us to convert the old sklearn object to the one that can run in Python 3.12. I will have a try today or tomorrow.

@bazarkua, do we has a branch with Python 3.12 so that once I have successfully completed this conversion, I can test it under this branch?

[bazarkua]

Thanks for taking look into this @chunyuma. Yes the migrated version code for ARAX running under python3.12 is under issue2521 branch. I will make more commits to the branch with specifying the exact versions with == like we had for old requirements.txt to ensure complete reproductivity.

[bazarkua]

Migration of ARAX from python3.9 to python3.12 is blocked for now, until @chunyuma can refactor xCRG model to work with scikit-learn==1.2.2 which is the oldest compatible version for python3.12

[saramsey]

OK, thank you for the update, @bazarkua. And thank you for carefully researching this issue.

[chunyuma]

Hi @bazarkua @bazarkua, I want to provide some updates: after testing and investigating almost all potential methods to convert model file from the old version to python3.12 with scikit-learn>=1.2.2, they all seem to fail due to the significant change in the internal data structures in newer version since scikit-learn version > 1.2. Thus, we still have to re-generate the models by fitting them again. I have already started working on this yesterday. Good news is that we just need to re-run scikit-learn to retrain the model because many previous intermediate files were saved on my disk. But it needs 2-3 days. Once I make it done, I will update the files under issue2521 branch.

[bazarkua]

Sounds good, thank you for the update @chunyuma!

[bazarkua]

"The Jaeger community deprecated their client libraries and recommends migrating to the OpenTelemetry APIs, SDKs and instrumentations." https://opentelemetry.io/docs/migration/

I've made some changes to migrate to the new OpenTelemetry because Jaeger client libraries deprecated July 2023. OpenTelemetry is now industry standard for observability. Supposedly we should get better performance, security, and maintainability.

Before: Multiple network hops (App -> Agent -> Collector -> Storage) After: Direct communication (App -> Collector -> Storage) OLD (Deprecated): JaegerExporter -> Thrift Protocol -> Port 6831 NEW (Modern): OTLPSpanExporter -> OTLP Protocol -> Port 4318

Python 3.12 + OpenTelemetry 1.35.0 sends OTLP data to port 4318, and modern Jaeger backends (v1.35+) can receive it Will be testing this on arax.rtx.ai and then on arax.ncats.io/test endpoint

[bazarkua]

All migrated code is isolated in the branch issue2521, needs a lot of testing to ensure successful migration of ARAX to Python 3.12. If migration to Python 3.12 is successful we might get performance boost, memory efficiency better concurrency, security and stability, and better debugging for future development of ARAX

[chunyuma]

Hi @bazarkua, I have re-trained xCRG, uploaded the new model files to arax-databases.rtx.ai and updated the config_dbs.json file. Now it can work with python 3.12 and scikit-learn==1.7.1. But it seems like currently Plover has something wrong and the relevant tests cannot pass.

[saramsey]

Well done, @chunyuma !

[bazarkua]

Moving forward with the safer approach and only upgrading packages that are not compatible with python3.12 scikit-learn==1.7.1 Upgraded from very old 1.1.2 (Thanks to @chunyuma for re-trained xCRG and uploading new model files) pandas==2.3.1 - Upgraded from 1.5.3 (not compatible with Python 3.12) numpy==1.26.4 - Upgraded from 1.24.2 (not compatible with Python 3.12) scipy==1.15.3 - Upgraded from 1.10.1 (not compatible with Python 3.12) PyYAML==6.0.2 - Upgraded from 6.0 (build issues on Python 3.12) lxml==4.9.4 - Upgraded from 4.9.1 (build issues on Python 3.12) pygit2==1.18.0 - Upgraded from 1.10.0 (build issues on Python 3.12) reasoner-validator==4.0.2 - Downgraded from 4.2.3 (version doesn't exist for python 3.12) pycryptodome==3.23.0 - Added (replaces incompatible simple-crypt) Removed asyncio==3.4.3 - Now built into Python 3.12

[edeutsch]

We do not want to downgrade reasoner-validator. We need the latest version in order to validate the latest TRAPI. I filed an issue with reasoner-validator:

https://github.com/NCATSTranslator/reasoner-validator/issues/105

[bazarkua]

Yes this makes sense, thanks for filling an issue with the reasoner @edeutsch!

[saramsey]

@bazarkua thank you for all your efforts to move this issue forward!

[edeutsch]

Version 5.0 of reasoner validator now formally supports Python 3.12: https://github.com/NCATSTranslator/reasoner-validator/releases

I have updated requirements.txt in master to include version 5.0 and run a pip install from requirements.txt on arax.ncats.io One snag:

The conflict is caused by:
    The user requested urllib3==1.26.19
    requests 2.32.0 depends on urllib3<3 and >=1.21.1
    reasoner-validator 5.0.0 depends on urllib3<3.0.0 and >=2.5.0

I tried upgrading urllib3 to a newer version (although ironically less than version 3) in an attempt to solve this.

This failed, however, since our version of boto3 appears to want a much older version.

The conflict is caused by:
    The user requested urllib3==2.5.0
    requests 2.32.0 depends on urllib3<3 and >=1.21.1
    botocore 1.27.95 depends on urllib3<1.27 and >=1.25.4

Seems like we'll need a much newer version of boto3. Maybe best to wait until the Python 3.12 upgrade to try to upgrade to the reasoner validator 5.0.0

[bazarkua]

Upgraded to the latest reasoner validator version to 5.0 (now that the latest version supports python3.12)

reasoner-validator 5.0.0 requires urllib3>=2.5.0 boto3 1.40.16 requires urllib3>=1.25.4,<3 (via botocore) requests 2.32.0 requires urllib3>=1.21.1,<3

The dependency conflict between reasoner-validator 5.0.0 and the older boto3 version has been resolved by upgrading the following packages:

• Upgraded boto3: 1.24.59 -> 1.40.16
• The newer version supports urllib3>=1.25.4,<3 which is compatible with reasoner-validator 5.0.0's requirement of urllib3==2.5.0
• Updated urllib3: 1.26.19 -> 2.5.0

This satisfies both reasoner-validator 5.0.0's requirement and the newer boto3's requirements

[bazarkua]

requirements.txt is now updated to be compatible with python3.12 and I was able to successfully install everything using pip install -r requirements.txt after purging pip cache and running on python 3.12:

((venv312) ) ubuntu@ip-172-31-18-95:~/RTX$ pip cache purge
Files removed: 961 (410.4 MB)

((venv312) ) ubuntu@ip-172-31-18-95:~/RTX$ which pip && python --version
/home/ubuntu/venv312/bin/pip
Python 3.12.11

((venv312) ) ubuntu@ip-172-31-18-95:~/RTX$ pip install -r requirements.txt 
Collecting pandas==2.3.1 (from -r requirements.txt (line 3))
  Downloading pandas-2.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (91 kB)
...
...
...
Successfully installed CacheControl-0.12.11 Flask-2.2.5 Flask-Cors-3.0.10 Jinja2-3.1.6 MarkupSafe-2.1.2 PyYAML-6.0.2 SQLAlchemy-1.4.47 Six-1.17.0 aiohttp-3.9.4 aiosignal-1.4.0 annotated-types-0.7.0 asttokens-3.0.0 atomicwrites-1.4.1 attrs-25.3.0 backoff-2.2.1 bioregistry-0.12.33 bmt-1.4.5 boto3-1.40.16 botocore-1.40.16 cattrs-25.1.1 certifi-2025.8.3 cffi-1.17.1 chardet-5.2.0 charset-normalizer-3.4.3 click-8.2.1 clickclick-20.10.2 connexion-2.14.1 contourpy-1.3.3 curies-0.10.19 cycler-0.12.1 decorator-5.2.1 deprecated-1.2.18 deprecation-2.1.0 dictdiffer-0.9.0 dnspython-2.7.0 email-validator-2.2.0 executing-2.2.0 fastobo-0.12.3 fonttools-4.59.1 fpyutils-2.2.1 frozenlist-1.7.0 googleapis-common-protos-1.56.2 graphviz-0.20.1 greenlet-3.2.4 grpcio-1.74.0 hbreader-0.9.1 idna-3.10 importlib-metadata-6.0.1 inflection-0.5.1 iniconfig-2.1.0 ipython-9.4.0 ipython-cypher-0.2.6 ipython-pygments-lexers-1.1.1 itsdangerous-2.2.0 jedi-0.19.2 jmespath-1.0.1 joblib-1.2.0 json-flattener-0.1.9 jsonasobj2-1.0.4 jsonschema-4.25.1 jsonschema-specifications-2025.4.1 kiwisolver-1.4.9 linkml-runtime-1.9.5 lockfile-0.12.2 lxml-4.9.4 matplotlib-3.10.5 matplotlib-inline-0.1.7 md-toc-8.1.4 more_click-0.1.2 msgpack-1.1.1 multidict-6.6.4 neo4j-driver-1.7.6 neo4jrestclient-2.1.1 neobolt-1.7.17 neotime-1.7.4 networkx-2.8.8 nltk-3.7 numpy-1.26.4 opentelemetry-api-1.17.0 opentelemetry-exporter-jaeger-1.17.0 opentelemetry-exporter-jaeger-proto-grpc-1.17.0 opentelemetry-exporter-jaeger-thrift-1.17.0 opentelemetry-exporter-otlp-proto-http-1.17.0 opentelemetry-instrumentation-0.38b0 opentelemetry-instrumentation-aiohttp-client-0.38b0 opentelemetry-instrumentation-flask-0.38b0 opentelemetry-instrumentation-requests-0.38b0 opentelemetry-instrumentation-wsgi-0.38b0 opentelemetry-proto-1.17.0 opentelemetry-sdk-1.17.0 opentelemetry-semantic-conventions-0.38b0 opentelemetry-util-http-0.38b0 packaging-25.0 pandas-2.3.1 parso-0.8.4 pexpect-4.9.0 pickledb-0.9.2 pillow-11.3.0 platformdirs-4.3.8 pluggy-1.6.0 poetry-core-2.1.3 prefixcommons-0.1.12 prefixmaps-0.2.6 prettytable-3.16.0 prompt_toolkit-3.0.51 pronto-2.5.3 propcache-0.3.2 protobuf-3.20.3 psutil-7.0.0 ptyprocess-0.7.0 pure-eval-0.2.3 pycparser-2.22 pycryptodome-3.23.0 pydantic-2.11.7 pydantic-core-2.33.2 pygit2-1.18.0 pygments-2.19.2 pymysql-1.1.1 pyparsing-3.2.3 pystow-0.7.7 pytest-7.2.2 pytest-logging-2015.11.4 python-dateutil-2.9.0.post0 pytrie-0.4.0 pytz-2025.2 rdflib-7.1.4 reasoner-validator-5.0.0 referencing-0.36.2 regex-2025.7.34 requests-2.32.0 requests-cache-1.2.1 rpds-py-0.27.0 s3transfer-0.13.1 scikit-learn-1.7.1 scipy-1.15.3 seaborn-0.13.2 setproctitle-1.3.3 setuptools-80.9.0 sortedcontainers-2.4.0 stack_data-0.6.3 stringcase-1.2.0 swagger-ui-bundle-0.0.9 tabulate-0.9.0 threadpoolctl-3.6.0 thrift-0.22.0 tomark-0.1.4 tornado-6.4.1 tqdm-4.67.1 traitlets-5.14.3 typing-extensions-4.14.1 typing-inspection-0.4.1 tzdata-2025.2 ujson-5.4.0 url-normalize-2.2.1 urllib3-2.5.0 wcwidth-0.2.13 werkzeug-2.3.8 wrapt-1.17.3 xgboost-1.7.6 yarl-1.20.1 zipp-3.23.0

[saramsey]

@bazarkua what's the status on this issue?

[bazarkua]

Hi @saramsey thank you for helping me to start on setting up Dockefile during our last meeting, I continued from where we left off with the progress we made on issue2521.rtx.ai I moved Docker file from there to arax.rtx.ai and I was able to migrate Dockerfile setup from Python 3.9 to Python 3.12.

I've created a new Dockerfile that uses Ubuntu 22.04 as the base image and installs Python 3.12.11 from the deadsnakes PPA. After building the Docker image, created a container that mounts the local RTX codebase as a volume, allowing it to use existing databases and configuration files.

Now on arax.rtx.ai the container successfully runs Apache2 as the web server and the Flask OpenAPI service with Python 3.12, with Apache2 configured to proxy API requests to the Flask service. I've verified the setup works by testing queries and it returns proper responses. This Docker container on arax.rtx.ai now closely replicates the production Docker and Apache2 setup

[bazarkua]

@edeutsch @saramsey I wanted to test issue2521 branch running on arax.ncats.io/test endpoint, is it okay if I install python3.12 onto rtx1 docker container and then create a venv with python3.12 to isolate test directory and running ARAX under python 3.12? It shouldn't disrupt other endpoint running services, since they have path to python3.9, and I'm not deleting python3.9 (System python3.9 will be unchanged as well as all existing python3.9 packages)

[edeutsch]

Hi @bazarkua, sorry for the delay. Yes, this sounds fine. As far as I'm aware all endpoints on arax.ncats.io run on the custom compiled 3.9 and thus should not be affected by installing 3.12.