torchx
torchx copied to clipboard
pytorch
Docker build
Added a docker API client, and changed the build function to its low-level version. This makes it return an event stream that we can then log to screen for real-time update on the image build process. This prints the same things as running docker build locally, so docker RUN commands can get quite verbose, we could filter the 'stream' events and only print the ones starting with 'Step ' or '---> ' .
Also changed the log message from api.py to make more sense (encompasses the case where dockerfile.torchx is used).
#813
- Fixed the pyre typing errors
- Unitest are failling because of codecov failing to upload?
- Slurm integration stuck in post
I also removed a redundant error case in my commit ( next() doesnt raise ValueError) and renamed output to build_msg.
My previous commit #809 was flawed, it didn't change logic cause if it did, it would introduce a bug. Ill fix it in a different commit, in the meantime this commit is in draft mode.
@ccharest93 are you interested in polishing this up and landing it? cc @clumsy
Ok i cleaned up the commit, i still need to rebuild my devcontainer to run tests and fix any errors that show up, will do this weekend. Would still like feedback on my original post though. Don't want to end up cluttering the torchx logs too much is my opinion.
This prints the same things as running docker build locally, so docker RUN commands can get quite verbose, we could filter the 'stream' events and only print the ones starting with 'Step ' or '---> ' .
@ccharest93 adding quiet scheduler_arg should help suppressing where it's needed.
