Make the edges related to input tensors

[junjun1023]

🚀 Feature

Hi, a differentiable input tensors (e.g. synthesized image) will get edges by calling Canny(low_threshold=xxxx, high_threshold=xxx), but the edges are not related to input tensor. It means I can't compute edge loss from synthesized images. Can you modify the way of implementation and not to break the relation of input image and its edge?

Motivation

I want to compute edge loss from synthesized image and real image, but the edges are requires_grad=False due to the current implementation of edge detection function.

Pitch

The edges of input synthesized images are able to compute loss and backward.

Alternatives

I've implemented a quite simple snippet based on your codes. But I'm not sure if multiplying [0, 1] will break the backward graph. The snippet below takes very much cuda memory after many epochs, I think it may due to while loop. Would you modify the current official implementation and make it less consume memory?

  # Threshold
  edges: torch.Tensor = F.threshold(magnitude, low_threshold, 0.0)
  edges = torch.where(edges > high_threshold, torch.tensor(1.0, dtype=torch.float, device=device), edges)
  edges = edges.to(dtype)

  if hysteresis:
      hysteresis_kernels: torch.Tensor = get_hysteresis_kernel(device, dtype)
      edges_old: torch.Tensor = -torch.ones(edges.shape, device=edges.device, dtype=dtype)

      while ((edges_old - edges).abs() != 0).any():
          weak: torch.Tensor = ((edges > 0.0) & (edges < 1.0)).float()
          weak_edge: torch.Tensor = edges * weak
          strong: torch.Tensor = (edges == 1).float()

          hysteresis_magnitude: torch.Tensor = F.conv2d(
              edges, hysteresis_kernels, padding=hysteresis_kernels.shape[-1] // 2
          ) # eight-direction

          hysteresis_magnitude = hysteresis_magnitude * (hysteresis_magnitude == 1).to(dtype)
          hysteresis_magnitude = torch.sum(hysteresis_magnitude, dim=1, keepdim=True)
          hysteresis_magnitude = hysteresis_magnitude * weak + strong

          edges_old = edges.clone()  
          edges = hysteresis_magnitude + (hysteresis_magnitude == 0) * weak * weak_edge

      edges = hysteresis_magnitude

Additional context

---

Consider also to contribute to Kornia universe projects :)

• Tutorials: our repository containing the tutorials.

[edgarriba]

@junjun1023 thanks for reporting. In order to check on differentiable, you must check that the following tests passes https://github.com/kornia/kornia/blob/master/test/filters/test_canny.py#L264

[liming-ai]

Hi @edgarriba,

Any updates on this issue? I also meet the same question that the edge cannot be differentiable

[edgarriba]

Possibly the algorithm needs a review and some tweaks. Currently we don’t have bandwidth and no priority to this one. Open to community to fix it.