[Bug] extract_non_reasoning_content error when predict is null

[simplew2011]

Prerequisite

• [x] I have searched Issues and Discussions but cannot get the expected help.
• [x] The bug has not been fixed in the latest version.

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

version 0.4.1

Reproduces the problem - code/configuration sample

from mmengine.config import read_base
from opencompass.models import OpenAISDK

with read_base():
    from opencompass.configs.datasets.math.math_500_gen import math_datasets



# pip install math_verify latex2sympy2_extended
# pip install langdetect
# opencompass eval_gsm8k.py
# opencompass opencompass/eval_ds3.py

datasets = math_datasets

from opencompass.utils.text_postprocessors import extract_non_reasoning_content

models = [
    dict(
        abbr='deepseek-r1-h800-32768',
        type=OpenAISDK,
        path='/root/.cache/huggingface',
        openai_api_base=['http://10.24.9.4:8000/v1'],
        # tokenizer_path='/opt/weights/DeepSeek-V3',
        key=
        'token-abc123',
        # meta_template=api_meta_template,
        # query_per_second=2,
        max_out_len=16384,
        max_seq_len=32768, # 32768
        temperature=0.6,
        batch_size=16,
        pred_postprocessor=dict(type=extract_non_reasoning_content),
        retry=10
        ),
]

Reproduces the problem - command or script

test math-500 using deepseek-r1 model

Reproduces the problem - error message

INFO 03-20 14:38:29 init.py:190] Automatically detected platform cuda. 03/20 14:38:30 - OpenCompass - INFO - Try to load the data from /home/wzp/.cache/opencompass/./data/math/ Traceback (most recent call last): File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 475, in inferencer.run() File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 86, in run self._score() File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 179, in _score pred_strs = [proc(s, **kwargs) for s in pred_strs] File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 179, in pred_strs = [proc(s, **kwargs) for s in pred_strs] File "/cx8k/fs100/wzp/code/opencompass/opencompass/utils/text_postprocessors.py", line 278, in extract_non_reasoning_content if think_start_token not in text and think_end_token in text: TypeError: argument of type 'NoneType' is not iterable

Other information

No response

[Redias]

How do you deploy the api server [http://10.24.9.4:8000/v1]? Can any-curl-command get the correct answer?

[liushz]

Can you show some responses without the pred_postprocessor.

[simplew2011]

How do you deploy the api server [http://10.24.9.4:8000/v1]? Can any-curl-command get the correct answer?

vllm serve /root/.cache/huggingface --tensor-parallel-size 16 --trust-remote-code --max-model-len 32768 --max-num-batched-tokens 32768 --enforce-eager --enable-reasoning --reasoning-parser deepseek_r1

[acylam]

Hi @simplew2011, you can troubleshoot by removing the pred_postprocessor and see if you get any predictions that way. If you do, it is likely due to the postprocessor function not extracting the content correctly. If you don't, check that your api server returns model responses as expected.