hail
hail copied to clipboard
[batch] Support uploading local files much like we upload a script file
What happened?
The hailtop.batch
client uploads a script file when the command becomes too large. This functionality frees users from thinking about the size of their commands. It's a great abstraction!
The client does not provide a mechanism to automatically upload local files or local directories which means that users must messily combine all their code and supporting data into one big file. For example,
local_in = hb.read_input('/path/to/local/script.sh')
local_data = hb.read_input('/path/to/small/local/reference.dat')
j = b.new_job()
j.command(f'bash {local_in} {local_data} {other_job.out_file} > {j.out_file}')
It need not necessarily be hb.read_input
, but it does seem like a good way to re-use an extant interface. hailtop.batch
should upload those files when it uploads large script files and then download them to the appropriate jobs.
Version
0.2.128
Relevant log output
No response