Fix broken gpu daily tests by creating the tmp_path directory

[j316chuck]

What does this PR do?

Fix broken gpu daily tests by creating the tmp_path directory

What issue(s) does this change relate to?

Before: https://github.com/mosaicml/composer/actions/runs/6385289342/job/17329711777 Pytorch 1.3

FAILED tests/callbacks/test_callbacks.py::TestCallbacks::test_multiple_fit_start_and_end[MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbacks::test_idempotent_close[MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbacks::test_idempotent_close[WandBLogger] - wandb.sdk.lib.mailbox.MailboxError: transport failed
FAILED tests/callbacks/test_callbacks.py::TestCallbacks::test_multiple_init_and_close[MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbacks::test_multiple_init_and_close[WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[1-False-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[1-False-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[2-False-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[2-False-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[1-True-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains[1-True-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[1-False-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[1-False-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[2-False-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[2-False-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[1-True-MLFlowLogger] - Exception: '/tmp/pytest-of-root/pytest-0/test_callback_is_constructable0/mlruns' does not exist.
FAILED tests/callbacks/test_callbacks.py::TestCallbackTrains::test_trains_multiple_calls[1-True-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[ActivationMonitor-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[CheckpointSaver-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[EarlyStopper-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[ExportForInferenceCallback-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[Generate-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[HealthChecker-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[ImageVisualizer-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[LRMonitor-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[MemoryMonitor-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[OptimizerMonitor-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[RuntimeEstimator-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[SpeedMonitor-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/callbacks/test_loggers_across_callbacks.py::test_loggers_on_callbacks[ThresholdStopper-WandBLogger] - wandb.errors.Error: An unexpected error occurred
FAILED tests/loggers/test_mlflow_logger.py::test_mlflow_experiment_init_unspecified - Exception: Invalid parent directory '/tmp/pytest-of-root/pytest-0/test_loggers_on_callbacks_Thre1/.trash'
FAILED tests/loggers/test_mlflow_logger.py::test_mlflow_experiment_init_experiment_name - Exception: Invalid parent directory '/tmp/pytest-of-root/pytest-0/test_loggers_on_callbacks_Thre1/.trash'
FAILED tests/loggers/test_mosaicml_logger.py::test_wandb_run_url - wandb.errors.Error: An unexpected error occurred
FAILED tests/loggers/test_wandb_logger.py::test_wandb_is_pickleable_when_disabled - wandb.errors.Error: An unexpected error occurred
ERROR tests/callbacks/test_image_visualizer.py::test_image_visualizer_with_wandb[9ba] - wandb.errors.Error: An unexpected error occurred
ERROR tests/callbacks/test_image_visualizer.py::test_image_visualizer_with_wandb[2ep] - wandb.errors.Error: An unexpected error occurred
ERROR tests/callbacks/test_image_visualizer.py::test_image_visualizer_with_wandb[7ep] - wandb.errors.Error: An unexpected error occurred
ERROR tests/callbacks/test_image_visualizer.py::test_image_visualizer_segmentation_with_wandb - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_log_image - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_ml_log_image_errors_out[images0-False] - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_ml_log_image_errors_out[images1-False] - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_ml_log_image_errors_out[images2-False] - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_ml_log_image_errors_out[images3-True] - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_log_image_with_masks - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_log_image_with_masks_and_table[images0-masks0] - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_log_table - wandb.errors.Error: An unexpected error occurred
ERROR tests/loggers/test_wandb_logger.py::test_wandb_log_metrics - wandb.errors.Error: An unexpected error occurred
= 34 failed, 3177 passed, 303 skipped, 916 deselected, 51 xfailed, 6 xpassed, 992 warnings, 13 errors in 649.69s (0:10:49) =

Distributed torch 2.0

tests/trainer/test_fsdp_checkpoint.py::test_fsdp_partitioned_state_dict_load[True-True-amp_bf16-sharded-adamw-False-2] 
----------End global rank 1 STDOUT----------
----------Begin global rank 1 STDERR----------
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
Fatal Python error: Aborted

Thread 0x00007f8b4ebcc700 (most recent call first):
  File "/composer/composer/loggers/remote_uploader_downloader.py", line 433 in _enqueue_uploads
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

After: WIP all tests pass ✅