Batch scoring fails with "DriverException: Job failed with There is no succeeded mini batch item returned from run()."

[sjuratov]

trafficstars

Here is what I see in DevOps

When I click on the error, I see that issue is with "Invoke Batch Scoring pipeline". However, I don't see any real issues here

In ml.azure.com, I see following

I also see following in one of the logs:

2020-09-17 09:43:39,318|ERROR|230|Failed to process {"id": 0, "dequeue_count": 3, "minibatch": [0], "location": "diabetes_scoring_ds", "partition_size": 1048576, "task_type": 10} with error The run() method in the entry script should return a list or a pandas DataFrame. The length of returned result is considered as succeeded number of items. For --output_action=append_row, the returned result is what to write into the result file. The actual type of output is NoneType. Please check logs for error. You can check logs/readme.txt for the layout of logs.. Detail: Traceback (most recent call last): File "/mnt/batch/tasks/shared/LS_root/jobs/*/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/mounts/workspaceblobstore/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/driver/azureml_user/parallel_run/payload_thread.py", line 137, in process_tasks self.process_task(task) File "/mnt/batch/tasks/shared/LS_root/jobs/*/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/mounts/workspaceblobstore/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/driver/azureml_user/parallel_run/payload_thread.py", line 69, in process_task score_output = processor.process_mini_batch(inputs) File "/mnt/batch/tasks/shared/LS_root/jobs/*/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/mounts/workspaceblobstore/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/driver/azureml_user/parallel_run/task_processor.py", line 96, in process_mini_batch self.validate(score_output) File "/mnt/batch/tasks/shared/LS_root/jobs/*/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/mounts/workspaceblobstore/azureml/4a3273ca-0a6e-4c0d-9168-e173bd2c34cb/driver/azureml_user/parallel_run/task_processor.py", line 38, in validate raise exc azureml_common.parallel_run.exception.WrongTypeFromRunException: The run() method in the entry script should return a list or a pandas DataFrame. The length of returned result is considered as succeeded number of items. For --output_action=append_row, the returned result is what to write into the result file. The actual type of output is NoneType. Please check logs for error. You can check logs/readme.txt for the layout of logs.

It seems to me like the problem is with run() method in parallel_batchscore.py but I can't figure it out.

Any ideas?

[j-so]

Have you made any customizations to parallel_batchscore.py or when training your model? Or are you hitting this error when running with the examples in this repo?

Batch prediction with AzureML requires that the run() method returns with an array or pandas Dataframe (doc).

Can you add some print lines in your run() method so we can see what is producing None?

[sjuratov]

Sorry for late reply. No, I am hitting this issue with example from repo.

I've tried to print pandas Dataframe but nothing gets printed.

[umeshp-de]

I am also facing the same issue with the as-is code from repos provided. Is anyone able to resolve this?

[lakpa-tamang9]

I also have similar issue while implementing the example. Any solutions yet?

[Arun-purakkatt]

I am also facing similar error .. any solutions yet ?

azureml_common.parallel_run.exception.NoResultToAppendError: There is no succeeded mini batch item returned from run()