nebuly
nebuly copied to clipboard
Add Faster Transformer compiler for Bert
WIP for #154
- [x] Code to install FasterTransformer
- [x] Code to optimize model using FasterTransformer
test this PR locally
WIP for #154
- [x] Code to install FasterTransformer
- [x] Code to optimize model using FasterTransformer
test this PR locally
- use the docker image (I'm not sure if the latest I have is actually the latest, it's id is 9e148a3a1d4f)
docker run --rm --gpus all -ti nebulydocker/nebullvm:latest
install FasterTransformer
git clone <repo>
cd nebullvm
git checkout <branch>
# install speedster and nebullvm separately
# because speedster depends on nebullvm, it may try to install from pypi?
pip install -e apps/accelerate/speedster
pip install -e .
# probably need to checkout to another directory before running the below command
# so faster_transformer's library code is installed to the correct location
python -m nebullvm.installers.auto_installer --f torch --compilers faster_transformer
- verify installation is successful
from nebullvm.operations.optimizations.compilers.utils import faster_transformer_is_available
faster_transformer_is_available()
- run the bert example
python notebooks/speedster/huggingface/faster_transformer_bert.py