Oceananigans.jl icon indicating copy to clipboard operation
Oceananigans.jl copied to clipboard
verified publisher icon CliMA

`no method matching fill_halo_kernel!` error when building `Fields` with explicit boundary conditions

Open tomchor opened this issue 4 months ago • 4 comments

The following fails on main, possibly due to the recently refactor:

julia> using Oceananigans

julia> grid = RectilinearGrid(topology=(Bounded, Bounded, Bounded), size=(4, 4, 4), extent=(1, 1, 1));

julia> boundary_conditions = FieldBoundaryConditions(top = GradientBoundaryCondition(1));

julia> b = CenterField(grid; boundary_conditions)
ERROR: MethodError: no method matching fill_halo_kernel!(::Oceananigans.BoundaryConditions.BottomAndTop, ::Oceananigans.BoundaryConditions.DefaultBoundaryCondition{…}, ::RectilinearGrid{…}, ::Symbol, ::Tuple{…}, ::OffsetArrays.OffsetArray{…}, ::Tuple{})

Closest candidates are:
  fill_halo_kernel!(::Oceananigans.BoundaryConditions.BottomAndTop, ::BoundaryCondition{<:Oceananigans.BoundaryConditions.MultiRegionCommunication}, ::Any, ::Any, ::Any, ::Any, ::Any)
   @ Oceananigans ~/repos/Oceananigans.jl/src/BoundaryConditions/fill_halo_kernels.jl:162
  fill_halo_kernel!(::Oceananigans.BoundaryConditions.BottomAndTop, ::Oceananigans.BoundaryConditions.DistributedCommunicationBoundaryCondition, ::Any, ::Any, ::Any, ::Any, ::Any)
   @ Oceananigans ~/repos/Oceananigans.jl/src/BoundaryConditions/fill_halo_kernels.jl:148
  fill_halo_kernel!(::Oceananigans.BoundaryConditions.BottomAndTop, ::BoundaryCondition{<:Oceananigans.BoundaryConditions.Periodic}, ::Any, ::Any, ::Any, ::Any, ::Any)
   @ Oceananigans ~/repos/Oceananigans.jl/src/BoundaryConditions/fill_halo_kernels.jl:132
  ...

Stacktrace:
  [1] fill_halo_kernels
    @ ~/repos/Oceananigans.jl/src/BoundaryConditions/fill_halo_kernels.jl:54 [inlined]
  [2] construct_boundary_conditions_kernels(bcs::FieldBoundaryConditions{…}, data::OffsetArrays.OffsetArray{…}, grid::RectilinearGrid{…}, loc::Tuple{…}, indices::Tuple{…})
    @ Oceananigans.BoundaryConditions ~/repos/Oceananigans.jl/src/BoundaryConditions/fill_halo_kernels.jl:19
  [3] #construct_regionally#82
    @ ~/repos/Oceananigans.jl/src/Utils/multi_region_transformation.jl:159 [inlined]
  [4] construct_regionally
    @ ~/repos/Oceananigans.jl/src/Utils/multi_region_transformation.jl:154 [inlined]
  [5] macro expansion
    @ ~/repos/Oceananigans.jl/src/Utils/multi_region_transformation.jl:244 [inlined]
  [6] (Field{…})(grid::RectilinearGrid{…}, data::OffsetArrays.OffsetArray{…}, bcs::FieldBoundaryConditions{…}, indices::Tuple{…}, op::Nothing, status::Nothing, buffers::Nothing)
    @ Oceananigans.Fields ~/repos/Oceananigans.jl/src/Fields/field.jl:35
  [7] Field
    @ ~/repos/Oceananigans.jl/src/Fields/field.jl:106 [inlined]
  [8] Field(loc::Tuple{…}, grid::RectilinearGrid{…}, T::DataType; indices::Tuple{…}, data::OffsetArrays.OffsetArray{…}, boundary_conditions::FieldBoundaryConditions{…}, operand::Nothing, status::Nothing)
    @ Oceananigans.Fields ~/repos/Oceananigans.jl/src/Fields/field.jl:196
  [9] CenterField
    @ ~/repos/Oceananigans.jl/src/Fields/field.jl:208 [inlined]

tomchor avatar Sep 03 '25 19:09 tomchor

I think you are not meant to use the FieldBoundaryConditions like that, but specifying a grid and a location always. Also on version 0.97.8

julia> grid = RectilinearGrid(topology=(Bounded, Bounded, Bounded), size=(4, 4, 4), extent=(1, 1, 1));

julia> bcs = boundary_conditions = FieldBoundaryConditions(top = GradientBoundaryCondition(1));

julia> b = CenterField(grid; boundary_conditions)
4×4×4 Field{Center, Center, Center} on RectilinearGrid on CPU
├── grid: 4×4×4 RectilinearGrid{Float64, Bounded, Bounded, Bounded} on CPU with 3×3×3 halo
├── boundary conditions: FieldBoundaryConditions
│   └── west: Default, east: Default, south: Default, north: Default, bottom: Default, top: Gradient, immersed: Default
└── data: 10×10×10 OffsetArray(::Array{Float64, 3}, -2:7, -2:7, -2:7) with eltype Float64 with indices -2:7×-2:7×-2:7
    └── max=0.0, min=0.0, mean=0.0

julia> b.boundary_conditions
Oceananigans.FieldBoundaryConditions, with boundary conditions
├── west: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)
├── east: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)
├── south: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)
├── north: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)
├── bottom: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)
├── top: GradientBoundaryCondition: 1
└── immersed: DefaultBoundaryCondition (FluxBoundaryCondition: Nothing)

julia> Oceananigans.BoundaryConditions.fill_halo_regions!(b)
ERROR: MethodError: no method matching _fill_bottom_halo!(::Int64, ::Int64, ::RectilinearGrid{…}, ::OffsetArrays.OffsetArray{…}, ::Oceananigans.BoundaryConditions.DefaultBoundaryCondition{…}, ::Tuple{…})

Closest candidates are:
  _fill_bottom_halo!(::Any, ::Any, ::Any, ::Nothing, ::Any...)
   @ Oceananigans ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions_nothing.jl:16
  _fill_bottom_halo!(::Any, ::Any, ::Any, ::Any, ::Nothing, ::Any...)
   @ Oceananigans ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions_nothing.jl:14
  _fill_bottom_halo!(::Any, ::Any, ::Any, ::Any, ::BoundaryCondition{<:Oceananigans.BoundaryConditions.Open{<:Oceananigans.BoundaryConditions.FlatExtrapolation}}, ::Any, ::Any, ::Any)
   @ Oceananigans ~/development/Oceananigans.jl/src/BoundaryConditions/flat_extrapolation_open_boundary_matching_scheme.jl:122
  ...

Stacktrace:
  [1] macro expansion
    @ ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions.jl:216 [inlined]
  [2] cpu__fill_bottom_and_top_halo!
    @ ~/.julia/packages/KernelAbstractions/lGrz7/src/macros.jl:306 [inlined]
  [3] __thread_run(tid::Int64, len::Int64, rem::Int64, obj::KernelAbstractions.Kernel{…}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{…}, args::Tuple{…}, dynamic::KernelAbstractions.NDIteration.DynamicCheck)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/lGrz7/src/cpu.jl:145
  [4] __run(obj::KernelAbstractions.Kernel{…}, ndrange::Nothing, iterspace::KernelAbstractions.NDIteration.NDRange{…}, args::Tuple{…}, dynamic::KernelAbstractions.NDIteration.DynamicCheck, static_threads::Bool)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/lGrz7/src/cpu.jl:112
  [5] (::KernelAbstractions.Kernel{…})(::OffsetArrays.OffsetArray{…}, ::Vararg{…}; ndrange::Nothing, workgroupsize::Nothing)
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/lGrz7/src/cpu.jl:46
  [6] (::KernelAbstractions.Kernel{…})(::OffsetArrays.OffsetArray{…}, ::Vararg{…})
    @ KernelAbstractions ~/.julia/packages/KernelAbstractions/lGrz7/src/cpu.jl:39
  [7] _launch!(::CPU, ::RectilinearGrid{…}, ::Oceananigans.Utils.KernelParameters{…}, ::Function, ::OffsetArrays.OffsetArray{…}, ::Oceananigans.BoundaryConditions.DefaultBoundaryCondition{…}, ::Vararg{…}; exclude_periphery::Bool, reduced_dimensions::Tuple{}, active_cells_map::Nothing)
    @ Oceananigans.Utils ~/development/Oceananigans.jl/src/Utils/kernel_launching.jl:376
  [8] _launch!
    @ ~/development/Oceananigans.jl/src/Utils/kernel_launching.jl:363 [inlined]
  [9] launch!
    @ ~/development/Oceananigans.jl/src/Utils/kernel_launching.jl:340 [inlined]
 [10] fill_bottom_and_top_halo!(::OffsetArrays.OffsetArray{…}, ::Oceananigans.BoundaryConditions.DefaultBoundaryCondition{…}, ::BoundaryCondition{…}, ::Symbol, ::Tuple{…}, ::Tuple{…}, ::CPU, ::RectilinearGrid{…}; only_local_halos::Bool, kwargs::@Kwargs{…})
    @ Oceananigans.BoundaryConditions ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions.jl:341
 [11] fill_halo_event!(::OffsetArrays.OffsetArray{…}, ::Function, ::Tuple{…}, ::Tuple{…}, ::Tuple{…}, ::CPU, ::RectilinearGrid{…}; async::Bool, kwargs::@Kwargs{…})
    @ Oceananigans.BoundaryConditions ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions.jl:79
 [12] fill_halo_regions!(::OffsetArrays.OffsetArray{…}, ::FieldBoundaryConditions{…}, ::Tuple{…}, ::Tuple{…}, ::RectilinearGrid{…}; fill_boundary_normal_velocities::Bool, kwargs::@Kwargs{…})
    @ Oceananigans.BoundaryConditions ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions.jl:63
 [13] fill_halo_regions!
    @ ~/development/Oceananigans.jl/src/BoundaryConditions/fill_halo_regions.jl:50 [inlined]
 [14] #fill_halo_regions!#65
    @ ~/development/Oceananigans.jl/src/Fields/field.jl:810 [inlined]
 [15] fill_halo_regions!(::Field{…})
    @ Oceananigans.Fields ~/development/Oceananigans.jl/src/Fields/field.jl:807
 [16] top-level scope
    @ REPL[6]:1
Some type information was truncated. Use `show(err)` to see complete types.

This is because bcs are not regularized. However, we could support the pattern that you are suggesting by discarding the DefaultBoundaryCondition in favor of the actual boundary conditions when passing them into a field.

simone-silvestri avatar Sep 03 '25 22:09 simone-silvestri

There are two different ways to solve it:

  • make validate_boundary_conditions materialize the default bc into the actual bc
  • make construct_boundary_conditions_kernels materialize the default bc into the actual bc

because both the functions have info about the grid and location. I probably prefer the first one. Tomorrow I will open a PR to fix this.

simone-silvestri avatar Sep 03 '25 22:09 simone-silvestri

Ah, I wasn't aware this wasn't the proper usage. It seemed like it should work so I ended up posting it here. I'm okay with whatever solution you think is best. Thanks!

tomchor avatar Sep 03 '25 22:09 tomchor

Hmm, it is interesting, because we do have both the grid and location inside the Field constructor. We could possibly fix this by calling regularize_field_boundary_conditions inside the Field constructor?

glwagner avatar Sep 03 '25 23:09 glwagner