JetStream icon indicating copy to clipboard operation
JetStream copied to clipboard

Clean up Model Conversion Script

Open yeandy opened this issue 1 year ago • 2 comments

Currently the model conversion script will create a bucket export MODEL_BUCKET=gs://${USER}-maxtext. However, it may be the case that the gs://${USER}-maxtext path already exists, which I imagine would break the script.

Solution: Be able to read in a few more arguments MODEL_BUCKET and BASE_OUTPUT_DIRECTORY. We should also delete references to DATASET_PATH.

yeandy avatar Aug 16 '24 21:08 yeandy

If the bucket exists, the script will continue and use the existing ones IIRC. But feel free to refactor it to improve UX.

JoeZijunZhou avatar Aug 16 '24 23:08 JoeZijunZhou

If the bucket exists, the script will continue and use the existing ones IIRC

Yes, but only if the current USER is the original creator/owner of bucket, right? A different user could have the same value for USER, which I think would break the workflow.

yeandy avatar Aug 19 '24 17:08 yeandy