Error when using AutoTunning

[BlackFireAlex]

I used the tutorial to try fitting but I get an os error:

Steps to reproduce the behavior: Paste the code from the doc and run locally

I get this error:

(train_tune pid=18136) Exception in thread Thread-4:
(train_tune pid=18136) Traceback (most recent call last):
(train_tune pid=18136)   File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
(train_tune pid=18136)     self.run()
(train_tune pid=18136)   File "C:\Users\user\PycharmProjects\neural_forecast\venv\lib\site-packages\tensorboard\summary\writer\event_file_writer.py", line 233, in run
(train_tune pid=18136)     self._record_writer.write(data)
(train_tune pid=18136)   File "C:\Users\user\PycharmProjects\neural_forecast\venv\lib\site-packages\tensorboard\summary\writer\record_writer.py", line 40, in write
(train_tune pid=18136)     self._writer.write(header + header_crc + data + footer_crc)
(train_tune pid=18136)   File "C:\Users\user\PycharmProjects\neural_forecast\venv\lib\site-packages\tensorboard\compat\tensorflow_stub\io\gfile.py", line 761, in write
(train_tune pid=18136)     self.fs.write(self.filename, file_content, self.binary_mode)
(train_tune pid=18136)   File "C:\Users\user\PycharmProjects\neural_forecast\venv\lib\site-packages\tensorboard\compat\tensorflow_stub\io\gfile.py", line 150, in write
(train_tune pid=18136)     self._write(filename, file_content, "wb" if binary_mode else "w")
(train_tune pid=18136)   File "C:\Users\user\PycharmProjects\neural_forecast\venv\lib\site-packages\tensorboard\compat\tensorflow_stub\io\gfile.py", line 164, in _write
(train_tune pid=18136)     with io.open(filename, mode, encoding=encoding) as f:
(train_tune pid=18136) FileNotFoundError: [Errno 2] No such file or directory: b'C:\\Users\\user\\ray_results\\train_tune_2023-03-30_12-02-39\\train_tune_01490_00000_0_batch_size=256,input_size=240,learning_rate=0.0026,max_steps=1000,n_freq_downsample=8_4_2_1_1,n_pool_kern_2023-03-30_12-02-45\\lightning_logs\\version_0\\events.out.tfevents.1680170572.PC.18136.0'

[kdgutier]

Hey @BlackFireAlex,

I think a quick fix for this is to reset the lightning logs folder:

Can you confirm?

[BlackFireAlex]

Unfortunately not, I tried on two machines with two clean installs and with clearing the ray folder. I should mention I'm using a venv

[kdgutier]

Would you be able to provide extra details to dig deeper?

• I suppose you are using Windows?
• Are you executing this Automatic Hyperparameter Tuning notebook ?

Here is a list of common ray issues: https://github.com/ray-project/ray/issues/9114

We are yet to include Microsoft in the automated tests, this issue could be a good start

[BlackFireAlex]

Yes I am using windows. I have just executed the notebook you sent I get the exact same error when calling fit. Issue is reproductible on 2 windows laptops.

[luchungi]

I am getting the same error trying to run this notebook locally with some amendments to the config used for autotuning: https://colab.research.google.com/github/Nixtla/neuralforecast/blob/main/nbs/examples/Forecasting_TFT.ipynb

I am also on windows. The ray_results folder is defaulting to my windows user folder instead of my current working directory in VS code where I run the notebook.

Running the versions below neuralforecast 1.4.0 ray 2.3.1 pytorch lightning 1.9.5 (using 2.0.0 gives another error: support for validation_epoch_end has been removed in v2.0.0)