Error for 5_Default_LightGBM

[dersteppenwolf]

Example code:

automl = AutoML(mode="Compete", results_path=output_path, optuna_time_budget=3600, total_time_limit=(60*60), explain_level=2)

Environment:

Databricks Runtime Version 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

Error message:

/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/supervised/base_automl.py in _update_errors_report(self, model_name, error_msg) 1137 f"\n\nPlease set a GitHub issue with above error message at: {link}" 1138 ) -> 1139 fout.write("\n\n") 1140 1141 def select_and_save_best(self, show_warnings=False): [Errno 95] Operation not supported OSError: [Errno 95] Operation not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/supervised/base_automl.py", line 1084, in _fit trained = self.train_model(params) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/supervised/base_automl.py", line 371, in train_model mf.train(results_path, model_subpath) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/supervised/model_framework.py", line 286, in train learner.save(p) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/supervised/algorithms/lightgbm.py", line 277, in save self.model.save_model(model_file_path) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/lightgbm/basic.py", line 3306, in save_model _dump_pandas_categorical(self.pandas_categorical, filename) File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-74e93b91-e9d9-4149-9830-8a905479fb90/lib/python3.8/site-packages/lightgbm/basic.py", line 623, in dump_pandas_categorical f.write(pandas_str) OSError: [Errno 95] Operation not supported

[pplonski]

Can you run the code in a non-databricks environment? Do you have a similar error?

[zwang2019]

OSError: [Errno 95] Operation not supported. This is because the idiot databrick doesn't support direct appends and random writes since Runtime 6.0 !!!!!

You have to save to a local disk, such as /tmp or something. After finish training, copy and paste to dbfs :(

check link below: https://docs.microsoft.com/en-us/azure/databricks/kb/dbfs/errno95-operation-not-supported

[pplonski]

thanks, @zwang2019 for the info!

Are you not using Databricks AutoML? I thought they had some?

[zwang2019]

thanks, @zwang2019 for the info!

Are you not using Databricks AutoML? I thought they had some?

Honestly speaking, this is one of the ways MS forces users to use their AutoML. I am just trying to deploy my work from Linux server to 'Cloud' which simply using LightGBM.

[pplonski]

May I ask how would you like to deploy? Are you going to use model as REST API or build some UI around it?