opencompass icon indicating copy to clipboard operation
opencompass copied to clipboard

[Bug] 🐛 Type Err in Humaneval

Open b2r66sun opened this issue 1 year ago • 0 comments

Prerequisite

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

{
   'CUDA available': True,
   'CUDA_HOME': '/usr/local/cuda-11.7',
   'GCC': 'gcc (GCC) 8.4.1 20200928 (Anolis 8.4.1-1.0.1)',
   'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-PCIE-40GB',
   'MMEngine': '0.10.3',
   'MUSA available': False,
   'NVCC': 'Cuda compilation tools, release 11.7, V11.7.99',
   'OpenCV': '4.9.0',
   'PyTorch': '1.13.1+cu117',
   'PyTorch compiling details': 'PyTorch built with:\n'
                                '  - GCC 9.3\n'
                                '  - C++ Version: 201402\n'
                                '  - Intel(R) Math Kernel Library Version '
                                '2020.0.0 Product Build 20191122 for Intel(R) 64 '
                                'architecture applications\n'
                                '  - Intel(R) MKL-DNN v2.6.0 (Git Hash '
                                '52b5f107dd9cf10910aaa19cb47f3abf9b349815)\n'
                                '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                                '  - LAPACK is enabled (usually provided by '
                                'MKL)\n'
                                '  - NNPACK is enabled\n'
                                '  - CPU capability usage: AVX2\n'
                                '  - CUDA Runtime 11.7\n'
                                '  - NVCC architecture flags: '
                                '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n'
                                '  - CuDNN 8.5\n'
                                '  - Magma 2.6.1\n'
                                '  - Build settings: BLAS_INFO=mkl, '
                                'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
                                'CUDNN_VERSION=8.5.0, '
                                'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                                'CXX_FLAGS= -fabi-version=11 -Wno-deprecated '
                                '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                                '-fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM '
                                '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                                '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                                '-DEDGE_PROFILER_USE_KINETO -O2 -fPIC '
                                '-Wno-narrowing -Wall -Wextra '
                                '-Werror=return-type -Werror=non-virtual-dtor '
                                '-Wno-missing-field-initializers '
                                '-Wno-type-limits -Wno-array-bounds '
                                '-Wno-unknown-pragmas -Wunused-local-typedefs '
                                '-Wno-unused-parameter -Wno-unused-function '
                                '-Wno-unused-result -Wno-strict-overflow '
                                '-Wno-strict-aliasing '
                                '-Wno-error=deprecated-declarations '
                                '-Wno-stringop-overflow -Wno-psabi '
                                '-Wno-error=pedantic -Wno-error=redundant-decls '
                                '-Wno-error=old-style-cast '
                                '-fdiagnostics-color=always -faligned-new '
                                '-Wno-unused-but-set-variable '
                                '-Wno-maybe-uninitialized -fno-math-errno '
                                '-fno-trapping-math -Werror=format '
                                '-Werror=cast-function-type '
                                '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                                'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                                'PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, '
                                'USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, '
                                'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, '
                                'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, '
                                'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n',
   'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
   'TorchVision': '0.14.1+cu117',
   'numpy_random_seed': 2147483648,
   'opencompass': '0.2.2+ae0858c',
   'sys.platform': 'linux'
}

Reproduces the problem - code/configuration sample

with read_base():
    from .datasets.humaneval.humaneval_gen import humaneval_datasets

datasets = [
    *humaneval_datasets,
]

Reproduces the problem - command or script

python run.py /path/to/config/config.py -m eval -r /path/to/timestamp/data

Reproduces the problem - error message

AttributeError: 'bool' object has no attribute 'lstrip'

Other information

Due to unexpected output format from LLM, This function humaneval_postprocess_v2 may raise some type errors while handling text = text.lstrip('\n').

def humaneval_postprocess_v2(text: str) -> str:
    """This is an advanced version of previous postprocess to handle more
    situations, better to use this one."""
    try:
        # for chatGLM raw text
        text = eval(text)
    except Exception:
        pass
    text = text.lstrip('\n')

e.g., While the output from LLM is only a 'False', the processing function will directly take it as a bool. Some type checks or convertions may be required

b2r66sun avatar Feb 27 '24 06:02 b2r66sun