Does ForwardDiff work with ImageTransformations?

[arbenede]

I am trying to use ForwardDiff to compute the gradient of a cost function defined as the Sum of Square Differences (SSD) between two images with respect to the parameters of an image transformation. In the code shown below, one of the images has been translated by a small amount. This is basically just a toy problem that I designed to understand if ForwardDiff.jl and ImageTransformations.jl play well together. Eventually, I will compute derivatives relative the parameters of different image warps. Notice that I do not want to compute gradients of the pixel intensities, but, say, the 2x2 Jacobian of the translation, like:

using Images
using TestImages
using ImageTransformations
using ForwardDiff

img1 = testimage("cameraman.tif")

y_shift = 0.33
x_shift = -1.76
t = ImageTransformations.Translation(y_shift, x_shift)

img2 = ImageTransformations.warp(img1, t, ImageTransformations.indices_spatial(img1), 0)

function cost(Δ::Vector{T}) where T <: Real
    t = ImageTransformations.Translation(Δ[2], Δ[1])
    img2 = ImageTransformations.warp(img1, t, ImageTransformations.indices_spatial(img1), 0)

    imgg1 = Gray.(img1)
    imgg2 = Gray.(img2)

    mat1 = convert(Array{Float64}, imgg1)
    mat2 = convert(Array{Float64}, imgg2)
    
    @. mat1 = (mat1 - mat2)^2
    SSD = sum(vec(mat1))
    
    return SSD
end

I’m getting this error:

StackOverflowError:

Stacktrace:
  [1] make_typealias(x::Type)
    @ Base ./show.jl:531
  [2] show_typealias(io::IOBuffer, x::Type)
    @ Base ./show.jl:661
  [3] show(io::IOBuffer, x::Type)
    @ Base ./show.jl:816
  [4] show_datatype(io::IOBuffer, x::DataType)
    @ Base ./show.jl:928
  [5] show(io::IOBuffer, x::Type)
    @ Base ./show.jl:819
  [6] print(io::IOBuffer, x::Type)
    @ Base ./strings/io.jl:35
  [7] print_to_string(::String, ::Vararg{Any, N} where N)
    @ Base ./strings/io.jl:135
  [8] string
    @ ./strings/io.jl:174 [inlined]
  [9] floattype
    @ ~/.julia/packages/FixedPointNumbers/HAGk2/src/deprecations.jl:4 [inlined]
 [10] _default_digits(#unused#::Type{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}) (repeats 43235 times)
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/autorange.jl:82
 [11] __round(x::StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/autorange.jl:100
 [12] (::ImageTransformations.var"#6#7"{Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}})(i::Int64)
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/autorange.jl:90
 [13] iterate
    @ ./generator.jl:47 [inlined]
 [14] _collect
    @ ./array.jl:691 [inlined]
 [15] collect_similar
    @ ./array.jl:606 [inlined]
 [16] map
    @ ./abstractarray.jl:2294 [inlined]
 [17] _round(tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/autorange.jl:89
 [18] _round(tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/autorange.jl:89
 [19] warp!(out::Matrix{Gray{N0f8}}, img::Interpolations.FilledExtrapolation{Gray{N0f8}, 2, Interpolations.BSplineInterpolation{Gray{N0f8}, 2, Matrix{Gray{N0f8}}, Interpolations.BSpline{Interpolations.Linear}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear}, Gray{N0f8}}, tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:92
 [20] warp(img::Interpolations.FilledExtrapolation{Gray{N0f8}, 2, Interpolations.BSplineInterpolation{Gray{N0f8}, 2, Matrix{Gray{N0f8}}, Interpolations.BSpline{Interpolations.Linear}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Linear}, Gray{N0f8}}, tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}, inds::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:88
 [21] warp(img::Matrix{Gray{N0f8}}, tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}, inds::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}, args::Int64; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:101
 [22] warp(img::Matrix{Gray{N0f8}}, tform::CoordinateTransformations.Translation{StaticArrays.SVector{2, ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}, inds::Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}, args::Int64)
    @ ImageTransformations ~/.julia/packages/ImageTransformations/xYRLH/src/warp.jl:100
 [23] cost(Δ::Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}})
    @ Main ./In[36]:18
 [24] vector_mode_dual_eval(f::typeof(cost), x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/QOqCN/src/apiutils.jl:37
 [25] vector_mode_gradient(f::typeof(cost), x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/QOqCN/src/gradient.jl:106
 [26] gradient(f::Function, x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}, ::Val{true})
    @ ForwardDiff ~/.julia/packages/ForwardDiff/QOqCN/src/gradient.jl:19
 [27] gradient(f::Function, x::Vector{Float64}, cfg::ForwardDiff.GradientConfig{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2, Vector{ForwardDiff.Dual{ForwardDiff.Tag{typeof(cost), Float64}, Float64, 2}}}) (repeats 2 times)
    @ ForwardDiff ~/.julia/packages/ForwardDiff/QOqCN/src/gradient.jl:17
 [28] top-level scope
    @ In[37]:1
 [29] eval
    @ ./boot.jl:360 [inlined]
 [30] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:1094

Any ideas?

[timholy]

It does not happen for me. What is your Julia version and all your package versions?

But I think you may have also omitted the final call?

[johnnychen94]

A similar issue is https://github.com/JuliaMath/Interpolations.jl/issues/430

@SomTambe has some recent experimental work in DiffImages.jl to bring Zygote support to warp, which might also be worth a try.

[timholy]

Ah, mostly I think we need https://github.com/JuliaGraphics/ColorTypes.jl/issues/131#issuecomment-824544538