nebuly icon indicating copy to clipboard operation
nebuly copied to clipboard

Add Faster Transformer compiler for Bert

Open cccntu opened this issue 2 years ago • 0 comments

WIP for #154

  • [x] Code to install FasterTransformer
  • [x] Code to optimize model using FasterTransformer

test this PR locally

WIP for #154

  • [x] Code to install FasterTransformer
  • [x] Code to optimize model using FasterTransformer

test this PR locally

  • use the docker image (I'm not sure if the latest I have is actually the latest, it's id is 9e148a3a1d4f)
docker run --rm --gpus all -ti nebulydocker/nebullvm:latest

install FasterTransformer

git clone  <repo>
cd nebullvm
git checkout <branch>

# install speedster and nebullvm separately
# because speedster depends on nebullvm, it may try to install from pypi?

pip install -e apps/accelerate/speedster
pip install -e .
# probably need to checkout to another directory before running the below command
# so faster_transformer's library code is installed to the correct location
python -m nebullvm.installers.auto_installer  --f torch --compilers faster_transformer
  • verify installation is successful
from nebullvm.operations.optimizations.compilers.utils import faster_transformer_is_available
faster_transformer_is_available()
  • run the bert example
python notebooks/speedster/huggingface/faster_transformer_bert.py

cccntu avatar Feb 03 '23 13:02 cccntu