Failed building t5 model in FastTransformer (Reached 82% then stopped)

[EmanElrefai12]

Branch/Tag/Commit

https://github.com/NVIDIA/FasterTransformer/blob/main/docs/t5_guide.md

Docker Image Version

nvcr.io/nvidia/pytorch:22.09-py3

GPU name

T4

CUDA Driver

12.0

Reproduced Steps

1. Run this command 
!git clone https://github.com/NVIDIA/FasterTransformer
2. Run this command 
!pip install -r FasterTransformer/examples/pytorch/t5/requirement.txt
3. Run this !apt-get -qq install cmake if you are using colab.
4. !cmake -DSM=70 -DCMAKE_BUILD_TYPE=Release -DBUILD_PYT=ON -DBUILD_MULTI_GPU=ON /content/FasterTransformer 
5. !make -j12 (the error here)

__________________________________________________


[ 82%] Built target th_common
/content/FasterTransformer/src/fastertransformer/utils/cuda_fp8_utils.cu(99): error: identifier "printf" is undefined

1 error detected in the compilation of "/content/FasterTransformer/src/fastertransformer/utils/cuda_fp8_utils.cu".
make[2]: *** [src/fastertransformer/utils/CMakeFiles/cuda_fp8_utils.dir/build.make:77: src/fastertransformer/utils/CMakeFiles/cuda_fp8_utils.dir/cuda_fp8_utils.cu.o] Error 2
make[1]: *** [CMakeFiles/Makefile2:2253: src/fastertransformer/utils/CMakeFiles/cuda_fp8_utils.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/content/FasterTransformer/src/fastertransformer/kernels/decoder_masked_multihead_attention_utils.h(1743): error: identifier "printf" is undefined

/content/FasterTransformer/src/fastertransformer/kernels/decoder_masked_multihead_attention_utils.h(1852): error: identifier "printf" is undefined

2 errors detected in the compilation of "/content/FasterTransformer/src/fastertransformer/kernels/unfused_attention_kernels.cu".
make[2]: *** [src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/build.make:77: src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/unfused_attention_kernels.cu.o] Error 2
make[1]: *** [CMakeFiles/Makefile2:3129: src/fastertransformer/kernels/CMakeFiles/unfused_attention_kernels.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

[rainyfly]

I met the same problem, any solution ?

[JihaoXin]

Please try to include <stdio.h> manually in two files mentioned at #770 .

[nacc]

Also hitting this in local testing via mlcommons/mlperf.