pynndescent icon indicating copy to clipboard operation
pynndescent copied to clipboard

Loading indices across various python versions

Open canergen opened this issue 11 months ago • 0 comments

Hi, this is more a developer question, then a bug report. I want to load classifiers like across different environments:

knn = make_pipeline(
    PyNNDescentTransformer(
        n_neighbors=self.classifier_dict["n_neighbors"],
        n_jobs=settings.n_jobs,
    ),
    KNeighborsClassifier(metric="precomputed", weights=self.classifier_dict["weights"]),
)
knn.fit(train_X, train_Y)
joblib.dump(
    knn,
    open(
        os.path.join(
            adata.uns["_save_path_trained_models"],
            "scvi_knn_classifier.joblib",
        ),
        "wb",
    ),
)

Currently they are restricted to a single python minor version (tested for python 3.10, 3.11 and 3.12). All expect different numbers of arguments during loading. The error is:

joblib stored in python 3.10 opened in python 3.12:    def _unpickle__CustomPickled(serialized):
        """standard unpickling for _CustomPickled.
    
        Uses NumbaPickler to load.
        """
>       ctor, states = loads(serialized)
E       TypeError: code() argument 13 must be str, not int

Do you have any suggestion to increase compatibility (when I disable numba.jit, it doesn't allow pickling the classifier)?

canergen avatar Jan 25 '25 04:01 canergen