AI-Hypercomputer
Fix "~/xpk" references
I tried making it `os.path.join(os.path.expanduser("~"), "directory")` in one of my earlier commits but that doesn't work. Explicitly someone made a mistake earlier on by not expanding `~` with [`os.path.expanduser`](https://docs.python.org/3/library/os.path.html#os.path.expanduser) so it actually expects in the maxtext directory a folder named `~` with the `xpk` folder inside. Otherwise the tests fail on CI.
Originally posted by @SamuelMarks in https://github.com/AI-Hypercomputer/maxtext/pull/1560#discussion_r2037944071
There are references to an unexpanded tilde in benchmark runner files specifically
benchmark_runner.py pw_long_running_recipe.py maxtext_xpk_runner.py pw_mcjax_benchmark_recipe.py pw_mcjax_checkpoint_benchmark_recipe.py
@SamuelMarks were there other references that you are aware of?
Yeah I register 9 files;rg -Fl 'xpk"':
- benchmarks/recipes/args_helper.py
- benchmarks/recipes/pw_mcjax_benchmark_recipe.py
- benchmarks/benchmark_runner.py
- benchmarks/recipes/pw_suspend_resume.py
- benchmarks/recipes/pw_elastic_training_recipe.py
- benchmarks/recipes/pw_mcjax_checkpoint_benchmark_recipe.py
- benchmarks/recipes/pw_long_running_recipe.py
- benchmarks/recipes/pw_remote_python_recipe.py
- benchmarks/maxtext_xpk_runner.py
I still count at least 7 relevant occurrences in main c32510d
I can send through a PR for this if you like; just didn't want to step on @bvandermoon's toes.
@SamuelMarks @lukebaumann what is the issue with the ~? Generally the runners look for xpk to be present in the users's home directory at ~/xpk. For example the benchmark runner is working properly currently
@bvandermoon Nothing is wrong with ~ but you can't just use it like that. That's why we see maxtext/~/xpk rather than /home/myname/xpk
$ python3 -c 'import os; print(os.path.expanduser("~"))'
/Users/myname
$ python3 -c 'import os; print("~")'
~
Details of os.path.expanduser: https://docs.python.org/3/library/os.path.html#os.path.expanduser
@bvandermoon there is an open PR as well, can we decide if any change is needed and close / merge accordingly ?
