Augmenting send list

[manavbhatia]

I am curious about the best way to add dof indices to the send-list on each rank. I am currently using the attach_extra_send_list_object function to attach derived object of AugmentSendList. This has been working fine for mesh without refinement.

Now that I use refinement, I am tripping this assert inside the call to reinit_send_list.

Searching through the code, I don't see anything using AugmentSendList, so I am curious if I should be using another API to get the same issue.

I suspect that this may not be enough information to dissect the issue, but I wanted to get some quick guidance before I invest in this further.

[roystgnr]

IMHO DofMap::add_algebraic_ghosting_functor() is preferred, if you can express your communication requirements as element coupling (possibly with sparse variable coupling) without losing much efficiency.

But AugmentSendList should still work, and should still be the preferred way to do things if you have more particular requirements that don't map cleanly onto linkages between elements.

Whatever's going on in your code isn't a problem with the send list, exactly - that assertion failure is finding a Node with an invalid processor_id()! And it's not anywhere near the send_list code, it's in a node constraints sync step...

Are you relying on libMesh node constraints? If not (e.g. you're not moving your mesh or you're not using adaptive refinement with hanging nodes or you're manually keeping hanging nodes consistent) then I'd say just stop configuring with --enable-nodeconstraint (or add --disable-nodeconstraint, if you're using --enable-everything).

It's barely possible that I just happened to fix this bug, in #2786, but I doubt it. If there's any way you could send me code that replicates the bug, I'd appreciate it.

[manavbhatia]

Thanks, Roy. I am using h-adaptivity with quad/hex elements. So, I am guessing that I will need to use enable-nodeconstraints (?). I will give the PR a shot. If things are not fixed, then I will try to get a standalone case setup.

[roystgnr]

Are you moving your mesh nodes, and relying on us to keep the hanging node locations consistent? --enable-nodeconstraints handles specifically that case. It's not necessary if you're not moving mesh nodes or if you're moving the hanging nodes consistently yourself, and it has nothing to do with DoF constraints.

[roystgnr]

Mind you, I'd still love a standalone case to debug, so we can fix whatever's wrong before someone who is in that boat needs the fix, I just don't want you to be stuck waiting unnecessarily.

[manavbhatia]

By moving, do you mean change the nodal coordinate, or move it from rank i to rank j on the communicator? My workflow is:

• initialize mesh/eq-sys/system
• Add extra dof_ids to send-list
• reinit
• analysis on initial mesh
• flag elements for refinement
• refine
• add new dof_ids to send-list and reinit system

This last step is throwing the exception.

[roystgnr]

I mean changing the nodal coordinate; whether you repartition or not doesn't affect whether the nodeconstraint code is needed.

[manavbhatia]

I see. I am not moving (nodal coordinates) any mesh points, but I am doing h-adaptivity. So if I understand correctly, I can turn off node-constraint in configuration?

Out of curiosity, if I don't change the mesh topology, why would movement of nodal locations require special handling? The constraints on hanging nodes, typically, for a Lagrange basis as based on simple interpolation along edges, correct? If so, then I am suspecting this interpolation changes once the mesh is moved and therefore requires updating the constraints?

[roystgnr]

So if I understand correctly, I can turn off node-constraint in configuration?

Right.

Out of curiosity, if I don't change the mesh topology, why would movement of nodal locations require special handling?

If you don't, then it doesn't. If you set your locations from DoFs in an already-properly-constrained Lagrange basis then no further constraints are necessary. But that's not the only way to move nodes; if you use only local information to move nodes then you're more likely than not to no longer have hanging nodes in a consistent position after a move, and this can correct for that.