how do you get the mask data?

[EmilyLike]

Hello, it's so luck to find the source code! I have a problem about the dataset. How do you get the mask data?

  		img1 = io.imread(path+"frame_"+num1+".png")

			img2 = io.imread(path+"frame_"+num2+".png")
			mask = io.imread(occpath+"frame_"+num1+".png")
			img1 = torch.from_numpy(transform.resize(img1, (360, 640))).to(device).permute(2, 0, 1).float()
			img2 = torch.from_numpy(transform.resize(img2, (360, 640))).to(device).permute(2, 0, 1).float()
			mask = torch.from_numpy(transform.resize(mask, (360, 640))).to(device).float()
			flow = read(flowpath+"frame_"+num1+".flo")

I have download the MPIDataset, and I don't find occlusion file. How do you get that?

[Yiman-GO]

There is occlusion data in MPI dataset.

[Yiman-GO]

And when I use this code, I got ridiculous results.The pictures are full of dots of different colors. I can't see any objects from the styled image. Have you got normal results？

[AntoineGerardeaux]

same as @Yiman-GO i got. Please help

[justanhduc]

@Yiman-GO @AntoineGerardeaux have you found any solution?

[AntoineGerardeaux]

not yet no

[xinix909]

@justanhduc @AntoineGerardeaux @Yiman-GO I fixed it by delete a "mul(255)" or something, there is more other bugs in the code. And now, still no temporal consictency. Flow wrap method is wrong I think.

[King-Hell]

@xinix909 Hello, you said "Flow wrap method is wrong I think." Did you mean that the warp(x,flo) function in utilities.py? I think the function may be used to calculate img1 from img2 rather than img2 from img1. Have you find any solution?

[safwankdb]

@justanhduc @AntoineGerardeaux @Yiman-GO I fixed it by delete a "mul(255)" or something, there is more other bugs in the code. And now, still no temporal consictency. Flow wrap method is wrong I think.

@xinix909 sorry I'm not maintaining this repo, can you please send a pull request for the fix?

[liulai]

@safwankdb @xinix909 @King-Hell @Yiman-GO @justanhduc @AntoineGerardeaux @EmilyLike hi, everyone，I have a solution! the MPIDataset have mask of img1 to img2, but it not enough, we need to block out all the intersections of two adjacent frames. So I compute the another mask

class MPIDataSet

class MPIDataSet(Dataset): def init(self, lines, train=True): self.lines = lines self.train = train

def __len__(self):
    return len(self.lines)

def __getitem__(self, idx):
    line = self.lines[idx]
    sample = line.strip().split(' ')
    img1 = Image.open(sample[0]).resize((weight, height), Image.BILINEAR)
    img2 = Image.open(sample[1]).resize((weight, height), Image.BILINEAR)

    img1 = ToTensor()(img1).float()
    img2 = ToTensor()(img2).float()

    if (self.train):
        mask = Image.open(sample[2]).resize((weight, height), Image.BILINEAR)
        flow = read(sample[3])
        mask = ToTensor()(mask).float()
        h, w, c = flow.shape
        flow = torch.from_numpy(transform.resize(flow, (height, weight))).permute(2, 0, 1).float()
        flow[0, :, :] *= float(flow.shape[1] / h)
        flow[1, :, :] *= float(flow.shape[2] / w)

        mask = 1 - mask
        mask[mask < 0.99] = 0
        mask[mask > 0] = 1
        return img1, img2, mask, flow
    else:
        return img1, img2

func warp

def warp(img, flow, device): b, c, h, w = img.size()

# mesh grid
x = torch.arange(0, w)
y = torch.arange(0, h)
y, x = torch.meshgrid(y, x)
gg = torch.cat((x.unsqueeze(0), y.unsqueeze(0))).repeat(b, 1, 1, 1).float().to(device)
gg = gg + flow

gg[:, 0, :, :] = 2.0 * gg[:, 0, :, :] / (w - 1) - 1.0
gg[:, 1, :, :] = 2.0 * gg[:, 1, :, :] / (h - 1) - 1.0
gg = gg.permute(0, 2, 3, 1)

output = torch.nn.functional.grid_sample(img, gg, mode='bilinear')
mask_boundary = torch.nn.functional.grid_sample(torch.ones(img.size(), device=device), gg, mode='bilinear')

mask_boundary[mask_boundary < 0.9999] = 0
mask_boundary[mask_boundary > 0] = 1

output = output * mask_boundary
return output, mask_boundary

This method cannot completely solve the problem due to error(light, luminance etc..), but it works well. you can see the result (video)

[safwankdb]

@liulai great work! Can you please raise a pull request for these changes?

[liulai]

@safwankdb Do you mean I upload another repo? I use your code

[safwankdb]

@liulai

@safwankdb Do you mean I upload another repo? I use your code

Pull Requests are the standard way for contributing to projects on GitHub. Here's a quick tutorial https://www.thinkful.com/learn/github-pull-request-tutorial/#Time-to-Submit-Your-First-PR.

However, if you don't want to go through the hassle, can you just email me a zip of your version at [email protected].

[liulai]

@safwankdb I pull the code to repo