axlearn icon indicating copy to clipboard operation
axlearn copied to clipboard

gcsfuse_mount_spec parsing and supporting mount_options from command line

Open deepikarajani24 opened this issue 5 months ago • 1 comments

This PR improves the command-line parsing for --gcsfuse_mount_spec to allow for multiple key-value pairs and robustly handle custom GCS FUSE mount_options.

Usage

  1. To specify custom mount_options: The entire argument must be wrapped in single quotes ('...') to protect the inner double quotes from the shell.
--gcsfuse_mount_spec='gcs_path=gs://my-bucket/data,mount_path=/data,mount_options="implicit-dirs,metadata-cache-ttl-secs=-1"'
  1. To use default mount_options: No special quoting is needed.
--gcsfuse_mount_spec=gcs_path=gs://my-bucket/data,mount_path=/data

Testing Unit Tests: Added parameterized unit tests to validate the new parsing logic for various formats (complex, basic, and empty mount_options).

End-to-End Test: Confirmed the full functionality by successfully launching a Fuji 7B training job that used a custom GCS FUSE mount defined with this flag.

The existing mount flags hardcoded in the code are not changed and I do see errors with those but haven't changed them. Instead passing the mount options I need in the command.

deepikarajani24 avatar Jul 11 '25 00:07 deepikarajani24

This pull request has been automatically marked as stale because it has been inactive for 60 days. It will be closed in 7 days if no further activity occurs. If you would like to continue working on this, please remove the stale label or leave a comment.

github-actions[bot] avatar Oct 19 '25 02:10 github-actions[bot]