How to use eval_tool?

[ftlong6666]

Hello @fyangneil could you explain the step how to do evaluation?,

1. should we convert all the predicted .png file into .mat?
2. How to setup your toolbox? I always got this error.

Error in crack_nms (line 11)
    load(fullfile(path_to_input, iids(i).name));

3. What should we do first? I tried to compile the toolbox successfully and modify the code into this.

mkdir(path_to_output);
iids = dir(fullfile(path_to_input, '*.png'));
for i = 1:length(iids)
    %edge = imread(fullfile(path_to_input, iids(i).name));\
    %load(fullfile(path_to_input, iids(i).name));
    %predmap=predmap;
    %disp(fullfile(path_to_input, iids(i).name))
    filename = fullfile(path_to_input, iids(i).name);
    predmap = imread(filename);
    predmap = rgb2gray(predmap);
    [Ox, Oy] = gradient2(convTri(predmap, 4));
    [Oxx, ~] = gradient2(Ox);
    [Oxy, Oyy] = gradient2(Oy);
    O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
    predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
    imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
    save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
end

However I got this error. Suppose that I have the predicted crack in .png file in the input_folder and I want to get the evaluation result only.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 17)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

[fyangneil]

@ftlong6666

1. you should convert to 'mat' file and refer to the results we uploaded to Google Drive or Baidu Yunpan.
2. You can use our dataset and provided results to test if the tool can work or not. If everything goes well, you should get the results reported in our paper.
3. Before using the tool, please make sure that in predicted crack map the bright regions are crack, background is black. To get ODS and OIS, run 'crack_nms.m' first then 'crack_eval.m'. To get AIU, run 'crack_AIU.m'..

[ftlong6666]

Hi I have tried it and I still got this error.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 14)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

Also how to change the path of Ground truth?, where path_to_input is the path of result.

[fyangneil]

@ftlong6666 Could you provide more details of how you use the tool. path_to_input is the directory to the crack prediction results (mat files).

[ftlong6666]

@fyangneil After I check I realized that the testdata is not in mat file. Here is the step to produce the error.

1. Extract your test data and fphb result
2. Change the path_to_input to the fphb location (the result in mat)
3. run the crack_nms. However here is the things: I don't know how to point the program to ground-truh location, and your testing groundtruth has no .mat file, so I need to convert to mat first. Do you have a code to convert all of the .png files into mat? Thank you

[ftlong6666]

Also I confuse with your code in file crack_nms.m, in line 9 predmap=predmap; where the predmap is not assign first.

[fyangneil]

@to run crack_nms, you do not need the ground truth files, only crack prediction files are needed.

[fyangneil]

@ftlong6666 the predmap is from line 8 "load(fullfile(path_to_input, iids(i).name))"

[ftlong6666]

@fyangneil Thank you for quick reply it helps a lot. I have changed it to predmap = load(fullfile(path_to_input, iids(i).name)); but I got this error.

Error using double
Conversion to double from struct is not possible.

Error in convn (line 27)
    A = double(A);

Error in convTri (line 93)
  J = convn(convn(J,f,'valid'),f','valid');

Error in crack_nms (line 15)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

[fyangneil]

@ftlong6666 you do not need to change the line 9, it should work.

[ftlong6666]

@fyangneil , I used exactly the same code here is the full code.

clear; clc;
path_to_input='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000';
path_to_output='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000/nms';
mkdir(path_to_output);
iids = dir(fullfile(path_to_input, '*.mat'));
for i = 1:length(iids)
    %edge = imread(fullfile(path_to_input, iids(i).name));\
    load(fullfile(path_to_input, iids(i).name));
    predmap=predmap;
    [Ox, Oy] = gradient2(convTri(predmap, 4));
    [Oxx, ~] = gradient2(Ox);
    [Oxy, Oyy] = gradient2(Oy);
    O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
    predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
    imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
    save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
end

But I still got this error message.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 10)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

/fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000 is the folder which contains your result.

[ftlong6666]

Here is the workspace.

[fyangneil]

@ftlong6666 what is the version of your matlab

[ftlong6666]

Matlab R2019b Windows 10

[fyangneil]

@ftlong6666 could you show the predmap?

[ftlong6666]

@fyangneil Here is the predmap variables.

[fyangneil]

@ftlong6666 looks good, it should be fine.Could you set a breakpoint in " J = convConst('convTri',I,r,s);"

[ftlong6666]

Sure here is the result. Also the content of variables. Here is the l.

[fyangneil]

@ftlong6666 have compiled the edges toolbox?

[ftlong6666]

Sure I have compiled it before.

[ftlong6666]

I do this one to compile.

Simply unzip, then add all directories to the Matlab path:
  >> addpath(genpath('/edges/toolbox/')); savepath;

If needed, run the compile script for the mex files:
  >> toolboxCompile;

[fyangneil]

Looks good. I have no idea

On Thu, Mar 5, 2020 at 12:53 PM ftlong6666 [email protected] wrote:

I do this one to compile.

Simply unzip, then add all directories to the Matlab path:

addpath(genpath('/edges/toolbox/')); savepath;

If needed, run the compile script for the mex files:

toolboxCompile;

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fyangneil/pavement-crack-detection/issues/10?email_source=notifications&email_token=ALAUWTTZOUA2TZ2WHSRODPDRF7RJHA5CNFSM4LA43X62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN6IBHI#issuecomment-595361949, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAUWTXXSV6M3LPOVJ7RW7LRF7RJHANCNFSM4LA43X6Q .

[ftlong6666]

1. What is your Matlab version?
2. Do you have a code to convert .png result into .mat?
3. Why we don't need the ground truth?, we should compare the result against the ground truth right?

[fyangneil]

For nms, you need groundtruth I do not have code to convert png to mat, you can do it using matlab. My matlab version is 2017b. But I think it has no business with the matlab version

On Thu, Mar 5, 2020 at 1:01 PM ftlong6666 [email protected] wrote:

1. What is your Matlab version?
2. Do you have a code to convert .png result into .mat?
3. Why we don't need the ground truth?, we should compare the result against the ground truth right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fyangneil/pavement-crack-detection/issues/10?email_source=notifications&email_token=ALAUWTTYMESA6JS4ITTH5GTRF7SJDA5CNFSM4LA43X62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN6I7YQ#issuecomment-595365858, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAUWTRIAEB722NWWUVJKLLRF7SJDANCNFSM4LA43X6Q .

[ftlong6666]

@fyangneil You mentioned that to run nms_crack.m we don't need ground-truth. So if we need it, in which part should I change the path? Also what is the dimension of the mat file?, is it in HxWx3 or HxWx1 or HxW?

[fyangneil]

@ftlong6666 for the deimension of mat file, you can check our provided results. When you run crack_eval.m and crack_AIU.m, you need ground truth files.

[ftlong6666]

@fyangneil If you don't have the code to convert .png to .mat, do you have the .mat of the groundtruth?, as long as you have the result, you should have the .mat of the groundtruth right?, in case you have, could you also upload it? Thank you

[ftlong6666]

Hello @fyangneil its just working, I don't know why. Anyway after I run the nms_crack.m , I try to run crack_eval.m . But I got this error, do you know why?

Index exceeds the number of array elements (0).

Error in edgesEvalDir_crack (line 68)
for i=1:n, id=ids{i}(1:end-4);idRes=idsRes{i}(1:end-4);

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Also in your code I see that you point it to directory testcrop folder, but I also see testdata folder, so which one is used? Also I realized that both of them don't have .mat files.

[fyangneil]

@ftlong6666 regarding crack_eval, only png image are needed. After nms, you can get png files, which used for evaluation. The ground truth file used for evaluation is also png files. The images in testcrop folder are the cropped chips from images in testdata. For details, you are referred to our paper (FPHBN).

[ftlong6666]

@fyangneil yes got it, but the problem is your code still looking for ".mat" files. First I put this line in edgesEvalImg.m in line 54 so the full code will be like this.

% load edges (E) and ground truth (G)
disp(E);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

Then after I run it, the result is still an error message like this.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_1281_1081.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 55)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 74)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Thus the error said that your code crack_eval.m is still looking for mat file. I am using your result that you provided in Google drive. Do you know what I am missing in here?, from the error I can see that your code try to load in image, but the path is still .mat .

[ftlong6666]

@fyangneil Are you sure use this code to provide the result?, because I just realize that the path of the result needs .png but the groundtruth should be in .mat. Let me show you. In the code edgesEvalDir.m line 70 you type this one.

im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);

but in edgesEvalImg.m line 53 if we print out the path, the E is the mat and G is the .png.

% load edges (E) and ground truth (G)
disp(E);
disp(G);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

so here is the result.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\testcrop2\20160222_081839_1281_1.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_115305_1921_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\testcrop2\20160222_163940_1_1081.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_641_1.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 56)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 75)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 6)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);
 

From here we can see that you try to read .mat files with imread, and read groundtruth .png with load which is wrong. If I change all of the GT to .mat files, there is no fields groundtruthand Boundaries. Yes this make sense that this code originally used to evaluate the edge detection from BSDS but since you have ground truth by your self, you should format it in the same field as in BSDS. Please could you help me how to do the evaluation?, I critically found that this is very crucial part, because its related with the comparison with another method. Thank you