Path normalization (train on Windows and deploy on Linux)

[ViacheslavDanilov]

I trained models on Windows, then I tried to use them on Linux, however, I could not load them due to an incorrect path joining. During model loading, I got learner_path in the following format experiments_dir/model_1/100_LightGBM\\learner_fold_0.lightgbm. The last two slashes were incorrectly concatenated with the rest part of the path. In this regard, I would suggest adding something like learner_subpath = learner_subpath.replace("\\", "/") before this code line. Though, there is a need to think about opposite cases: when a model is trained on Linux and then is used on Windows.

[pplonski]

@ViacheslavDanilov thank you for reporting the issue. The original path should be split based on \ and / and then joined with os.path.join - what do you think?

[ViacheslavDanilov]

@pplonski thanks for the prompt reply.

When I replace the double slash in learner_subpath, the learner_path variable is getting readable by l = AlgorithmFactory.load() and the code works well. As you see, os.path.join works only when a model is trained and tested on the same operating system i.e. trained on Windows and tested on Windows. However, when there are cases such as I encountered, the error is thrown. I would suggest normalizing the path somehow. The most straightforward solution I see is to use something like the following learner_subpath = learner_subpath.replace("\\", "/") if '\\' in learner_subpath else learner_subpath and call it right before learner_path = os.path.join(results_path, learner_subpath). At least, this resolves the problem with cross-platform usage.

Here is what I get using the source code:

results_path = 'experiments/LMN_compete_mae_Raw_Power_Source_0408_1405'
learner_subpath = '100_LightGBM\\learner_fold_0.lightgbm'
learner_path = 'experiments/LMN_compete_mae_Raw_Power_Source_0408_1405/100_LightGBM\\learner_fold_0.lightgbm'

[Karlheinzniebuhr]

Having the same issue when deploying a model from windows to linux, @ViacheslavDanilov were you able to solve it?

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/./mastermind/AI/ai.py", line 148, in predict
    return self.model.predict(features)[0]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/automl.py", line 387, in predict
    return self._predict(X)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/base_automl.py", line 1361, in _predict
    predictions = self._base_predict(X)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/base_automl.py", line 1318, in _base_predict
    X_stacked = self.get_stacked_data(X, mode="predict")
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/base_automl.py", line 421, in get_stacked_data
    oof = m.predict(X)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/ensemble.py", line 308, in predict
    y_predicted_from_model = model.predict(X)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/model_framework.py", line 429, in predict
    y_p = learner.predict(X_data)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/algorithms/lightgbm.py", line 270, in predict
    self.reload()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/algorithms/algorithm.py", line 36, in reload
    self.load(self.model_file_path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/supervised/algorithms/lightgbm.py", line 285, in load
    self.model = lgb.Booster(model_file=model_file_path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/lightgbm/basic.py", line 2639, in __init__
    _safe_call(_LIB.LGBM_BoosterCreateFromModelfile(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/lightgbm/basic.py", line 125, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: Could not open AutoML\4_Default_LightGBM_KMeansFeatures\learner_fold_0.lightgbm