ValueError: Classification metrics can't handle a mix of multiclass and unknown targets

[Vankhaido]

Hi swathikirans, I tried to apply this repository but I got a question:

After I run 'prepareGTEA61Dataset.py' script to make the dataset. I tried to evaluate the RGB model using the command

python eval-run-rgb.py --dataset gtea61 --datasetDir ./dataset/gtea_61/split2/test --modelStateDict best_model_state_rgb.pth --seqLen 25 --memSize 512

but I don't have any directory like ./dataset/gtea_61/split2/test so I changed it to ./gtea_dataset and use the following command to evaluate the model

python eval-run-rgb.py --dataset gtea61 --datasetDir ./gtea_dataset --modelStateDict best_model_state_dict_rgb_split2.pth --seqLen 25 --memSize 512

However, I got the error below when I did so:

Traceback (most recent call last): File "eval-run-rgb.py", line 91, in main() File "eval-run-rgb.py", line 89, in main main_run(dataset, model_state_dict, dataset_dir, seqLen, memSize) File "eval-run-rgb.py", line 58, in main_run cnf_matrix = confusion_matrix(true_labels, predicted_labels).astype(float) File "/home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 250, in confusion_matrix y_type, y_true, y_pred = _check_targets(y_true, y_pred) File "/home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 81, in _check_targets "and {1} targets".format(type_true, type_pred)) ValueError: Classification metrics can't handle a mix of multiclass and unknown targets

What did I do wrong then?

[swathikirans]

The test set of split2 consists of the videos of user S2. You can create a directory with name test and make a link to the directory S2 inside for creating the test set.

Is the error occurring only while computing the confusion matrix or you are not seeing the accuracy computed as well? I made a run with your setting and it is running as expected. So I am not sure what the source of your error.

[Vankhaido]

I tried to create a new directory named test and ran the command again python eval-run-rgb.py --dataset gtea61 --datasetDir ./gtea_dataset/gtea_61/split2/test --modelStateDict best_model_state_dict_rgb_split2.pth --seqLen 25 --memSize 512

but this was the result that I got:

Number of samples = 116 Evaluating... eval-run-rgb.py:49: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead. inputVariable = Variable(inputs.permute(1, 0, 2, 3, 4).cuda(), volatile=True) /home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/torch/nn/functional.py:1380: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead. warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.") /home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/torch/nn/functional.py:1368: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead. warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.") Test Accuracy = 0% Traceback (most recent call last): File "eval-run-rgb.py", line 91, in main() File "eval-run-rgb.py", line 89, in main main_run(dataset, model_state_dict, dataset_dir, seqLen, memSize) File "eval-run-rgb.py", line 58, in main_run cnf_matrix = confusion_matrix(true_labels, predicted_labels).astype(float) File "/home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 250, in confusion_matrix y_type, y_true, y_pred = _check_targets(y_true, y_pred) File "/home/james/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 81, in _check_targets "and {1} targets".format(type_true, type_pred)) ValueError: Classification metrics can't handle a mix of multiclass and unknown targets

So you could see that Test Accuracy = 0% and the error still popped up. I think the dataset that I downloaded could be the reason. I listed what I did below:

• Download RGB, Flow, Two Stream models
• Download the alternated frames and the corresponding warp optical flow of the GTEA 61 dataset
• Download GTEA dataset including VIDEOS, UNCOMPRESSED PNG FILES, HAND MASKS, ACTION LABELS (OLD)
• Change some lines of 'prepareGTEA61Dataset.py' script to match with downloaded dataset

4) label_dir = './gtea_labels_61/old_labels' # Dir containing the labels
5) src_img_dir = './gtea_png/png' # Dir containing the frames/flow
39) #change from "str(f).zfill(5)" to "str(f).zfill(10)", ".jpg" to ".png"
40) #change from "str(f).zfill(5)" to "str(f).zfill(10)", ".jpg" to ".png"

• Run prepareGTEA61Dataset.py
• Create a new directory path /gtea_dataset/gtea_61/split2/test/frames then copy S2 folder from /gtea61/frames to the new directory
• Evaluate the model using command python eval-run-rgb.py --dataset gtea61 --datasetDir ./gtea_dataset/gtea_61/split2/test --modelStateDict best_model_state_dict_rgb_split2.pth --seqLen 25 --memSize 512
• Then Error popped up

I was not sure if I needed to run the 'prepareGTEA61Dataset.py' script but I still ran it anyway.

[swathikirans]

I see that you are using PyTorchv0.4. I would suggest trying the code with PyTorch0.3.1.

[Vankhaido]

Hi swathikirans, thanks for your suggestion. Last few days I tried to create a new anaconda environment with Python 3.5 and Pytorch 0.3.1. However, I'm getting a new error

from ._ufuncs import * ImportError: /home/james/.local/lib/python3.5/site-packages/scipy/special/_ufuncs.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _gfortran_stop_numeric_f08

which doesn't seem to relate to ur code. I'm still trying to solve this error so I will report if I can apply your repo soon.

[RussellZab]

Hello!

I run prepareGTEA61Dataset.py with the settings: label_dir = '../data/GTEA/gtea_labels_61/old_labels' # Dir containing the labels src_img_dir = '../data/GTEA/gtea_flow' # Dir containing the frames/flow des_img_dir = './gtea_dataset' # Dir to which the images have to be copied

and, as output, have got 4 directories with frames: .gtea_dataset/S1, .gtea_dataset/S2, .gtea_dataset/S3, .gtea_dataset/S4.

Each of these 4 directories has subdirs with frames like 'close_coffee/1', 'close_honey/1', 'close_honey/2' and etc.

Then I tried command: python main-run-flow.py --dataset gtea61
--trainDatasetDir ./gtea_dataset/S2
--outDir experiments
--stackSize 5
--trainBatchSize 32
--numEpochs 750
--lr 1e-2
--stepSize 150 300 500
--decayRate 0.5

and got instantly the string as output 'Dir ./experiments/gtea61/flow exists!' But the dir ./experiments/gtea61/flow has empty files like train_log_acc.txt, train_log_loss.txt and etc.

So how I need to set --trainDatasetDir to run train-script correctly?

And pls add correct description at ReadMe.md because commands are not clear...

[swathikirans]

You can either remove the directory or run again with a different directory as output using the option "--outDir

".

For GTEA61, four train val splits are generally used. For split1, splits2-4 are used for training and split1 is used for validation. You can create a new directory for each train val split with the following structure: train_dir>flow_x>S2, S3, S4 train_dir>flow_y>S2, S3, S4

you can create it using symbolic links.

The prepareDataset.py file is for frames. I will try to update the git with detailed information on the ReadMe and all the required files that can perform these in the coming days