SCOREC
Add support for EGADS geometry system
This PR adds support for using the EGADS geometry system (acdl.mit.edu/ESP).
I've previously spoken to @cwsmith and @mortezah about the implementation, which is now fairly complete. Since this is my first fairly large PR to core, please let me know what else I should add to get this merged.
Hi @tuckerbabcock . Thank you for the contribution. I think pumi users will like having an open source cad option.
Can you add some test input files for use with test/split.c to the https://github.com/SCOREC/pumi-meshes repo and add a call to split with those inputs to test/testing.cmake within a conditional that checks if EGADS is enabled. Here is a similar example:
https://github.com/SCOREC/core/blob/d86c1ea4d3a7e2f27b733c578de52d6a2135b841/test/testing.cmake#L139-L146
Also, if possible, we should add a check on the version of EGADS that is found and compare that to a version range that we know works. Note, this is assuming EGADS provides version info somehow and it can be accessed at compile time. I can help with this as needed.
@cwsmith I'll work on adding a test that calls split with a mesh that uses an EGADS model. I'll also look into the version checks.
Thanks for your feedback!
@cwsmith the EGADS header files #define EGADSMAJOR and EGADSMINOR that hold the version number. Should I check the version in the cmake files so it's done as part of the configure step? I think I might need your help to implement that
Hi @cwsmith,
I've attempted to add tests for split, and added mesh and model files to the pumi-meshes repo under a folder named egads. The tests start to run on my machine, but fail because it can't find the model file, even though if I've verified that it's looking in the correct place and the model file does exist there. Any thoughts?
Thanks for taking a look at the changes @jacobmerson. Your last point about the allocation for the sup_filename is related to a design decision I made that probably deserves a little more scrutiny.
For models that look like this, a box and cylinder intersecting that share two internal faces along the intersection:
The egads topology functions will report 1 region for this model, even though there are conceptually two distinct regions (the cylinder part and the box part). Further, since egads only sees one region, it can not tell you what model faces are adjacent to the cylinder, or any other adjacency relationships that involve a region.
To address this, during the mesh generation process I've written out a model_name.egads.sup "supplementary" model file that contains the region adjacency information. This file will be read during gmi_egads_load and will allow the gmi_model to be queried about the region adjacency relationships.
I have access to the region adjacency information during mesh generation through the CAPS interface which is part of the Engineering Sketch Pad that EGADS itself is a part of. CAPS uses analysis interface modules (AIMS) to generate meshes. I've written an AIM that will construct a PUMI mesh from the internal CAPS mesh description. The internal mesh description contains the expected association between mesh volume elements and the model region they belong to, which I'm able to query to build the adjacency table that is written to model_name.egads.sup.
The gist of this is: EGADS does not internally keep track of regions -> I have manually kept track of model regions and their adjacency relationships (stored in the supplementary model file) -> I figure out these adjacency relationships during the mesh generation process since the CAPS internal mesh representation contains an association between volume elements and region ID.
Any thoughts @cwsmith, @jacobmerson?
I've made some changes that enable to use of EGADSlite, the HPC friendly version of EGADS. The main limitation of EGADSlite is that it cannot load an EGADS model on its own. To work with this I've implemented that rank 0 will use EGADS to load the model, and then broadcast the model over MPI to all other ranks that will import it with EGADSlite (based on an example in the EGADS distribution that follows this process). To have gmi_egads use either EGADS or EGADSlite based on the MPI rank I've switched from linking EGADS into the gmi_egads library, and instead load either EGADS or EGADSlite at runtime with dlopen.
A potential headache with this new approach is that I've had to hardcode the function prototypes for all the EGADS APIs I use as function pointers that will be loaded with the correct symbols at runtime using dlsym. This means that if EGADS changes the signature of any of these functions it will be a runtime error instead of a compile-time error. I think I could address this in the future by re-writing gmi_egads in C++ and using decltype and std::is_same: https://stackoverflow.com/questions/14541519/is-it-possible-to-write-c-template-macros-to-check-whether-two-functions-have/14541544#14541544
For those curious, here's the paper describing EGADSlite: https://acdl.mit.edu/ESP/Publications/AIAApaper2018-1401.pdf
Re: https://github.com/SCOREC/core/pull/330#issuecomment-779377675
I think as long as the process (and code) used to generate the supplementary info is available then we should be fine.
Re: egads model loading + bcast
Thanks for the paper link. From what I understand, this passage:
The initialization/model-read is oneof the differences, where anEGADSapplication prepares the data for use by the EGADSlite package. This data can be written to disk or used live in a parallel setting.
indicates that an EGADS API can be used to write a model that EGADSlite can load. Could a pumi utility be created that reads the supplementary info and egads model and writes the egadslite model? This seems like a simpler approach than broadcast + dlopen. As I understand building opencascade on a esoteric/non-x86 HPC system (i.e., Power9, A64x, etc.) may not be possible.
indicates that an EGADS API can be used to write a model that EGADSlite can load.
Yes, that's correct. The function EG_exportModel is used to write the loaded EGADS model into a stream. An EGADSlite instance would then call EG_importModel to load the model. I think it would be pretty easy to make a pumi utility that could handle this conversion. And you're right, just using EGADSlite and eliminating the dependency on opencascade would make this code much more portable.
I think as long as the process (and code) used to generate the supplementary info is available then we should be fine.
Yes, it's available here: https://github.com/tuckerbabcock/EngSketchPad/tree/beta
indicates that an EGADS API can be used to write a model that EGADSlite can load.
Yes, that's correct. The function
EG_exportModelis used to write the loaded EGADS model into a stream. An EGADSlite instance would then callEG_importModelto load the model. I think it would be pretty easy to make a pumi utility that could handle this conversion. And you're right, just using EGADSlite and eliminating the dependency on opencascade would make this code much more portable.
An idea... instead of making a separate pre-processing utility, would it be possible for ESP to write the EGADSLite file and the EGADS file (to support future modification of it in ESP) ?
I think a natural place to dump the EGADSlite model might be the preprocessing utility we already have that generates the PUMI mesh, the model file, and the supplementary model file. If we went with that route we could combine the EGADSlite file with the supplementary model into one file pretty easily too.
@cwsmith I've switched to just using EGADSlite, and no longer need the broadcast + dlopen process
I added back the option to build with EGADS instead of EGADSlite. In testing, I had an application that needed to use EGADS, and linking it with PUMI that used EGADSlite caused errors since EGADS and EGADSlite export the same symbols. Building the library defaults to using EGADSlite, but that can be adjusted with the option -DUSE_EGADSLITE=OFF.
@tuckerbabcock Thanks for the update. I think the petsc folks hit this problem so they created the following variant of EGADSlite with different function prefixes ("Original coding resulted in conflicts due to functions having the same name & signature"): https://github.com/bldenton/EGADSlite Would using their modified APIs make sense? There seems to be a bit of a risk to not using the 'official' MIT version...
My personal inclination is to stick with the "official" release, since it would allow better support should we need it. My other thought is that if one part of an executable needs to use EGADS for some reason, I would think that having PUMI also then revert to EGADS would not dramatically reduce performance (since whatever needed EGADS is likely to be the slowest part already), though is an untested assumption.
OK. Sounds good.
What is the performance concern with EGADS enabled vs EGADSlite enabled? Is it the broadcast of model info?
Jumping back to this comment: https://github.com/SCOREC/core/pull/330#issuecomment-808490780 I'd suggest having all ranks either use EGADS or EGADSlite. Hopefully this avoids the dlopen complexities. Were you already heading down this route with the latest change?
Note, the flag -DUSE_EGADSLITE should be prefixed with PUMI_ to avoid collisions with other libraries.
What is the performance concern with EGADS enabled vs EGADSlite enabled? Is it the broadcast of model info?
My comment was getting at that if an application needed EGADS it would need it because somewhere it would do more than just query the geometry (it would either creating or modifying it, which would be more expensive than a query). I was also thinking about the EGADSlite paper I linked above that shows that the EGADSlite functions are thread safe and scalable, and are in general much faster than the EGADS alternatives (Table 2 from that paper).
I'd suggest having all ranks either use EGADS or EGADSlite. Hopefully this avoids the dlopen complexities. Were you already heading down this route with the latest change?
Yes, that is the current state of the code. Based on the flag -DUSE_EGADSLITE (which I'll update to have the PUMI_ prefix), it will either compile against EGADS or EGADSlite.
@cwsmith which cmake changes are you thinking we should push off vs include with this pull request?
@cwsmith which cmake changes are you thinking we should push off vs include with this pull request?
I think we can skip the changes to variable names to include the PUMI prefix for now. The target_compile_definitions change is worth making now.
Hi @cwsmith: did you mean to assign this issue to me? Did you mean @tuckerbabcock? Tucker is our expert on all things EGADS.
Hi @cwsmith: did you mean to assign this issue to me? Did you mean @tuckerbabcock? Tucker is our expert on all things EGADS.
@jehicken Not intentionally. Sorry about that.
Thank you @cwsmith for the renewed interest in this PR.
Would you mind writing up something on the https://github.com/SCOREC/core/wiki/Mesh-Generation wiki page to discuss, or provide a reference to, how EGADS models and meshes can be used?
I'd be happy to write something like that up, but right now the mesh generation stuff isn't really polished at all. You have to use the ESP "analysis interface module" (AIM) I wrote for PUMI, with a slightly modified version of the ESP distribution (which I've been working on here: https://github.com/tuckerbabcock/EngSketchPad). Once you've got your mesh, you can use the gmi_egads interface I've written here with any ESP distribution. Long term, the mesh generation capabilities should not require using the modified ESP distribution (the AIM should be able to exist as a stand-alone plug-in), but I've set it up the way it is currently to make my life a little easier.
Besides the CMake changes discussed above, are there any other changes you intend to make? If not, I think we should merge this and resolve the CMake changes within the scope of a larger PUMI CMake refactor. @jacobmerson Does that make sense to you?
No, I think the gmi_egads interface is mostly feature complete as is right now (for everything I've needed to use it for anyway). Of course I occasionally find bugs while working on it, but those can easily be submitted as their own PRs in the future.
@tuckerbabcock do you have instructions for how to install/run the other necessary components? Does EGADS always require ESP? Unless I had read the comments on this thread I would have no idea what ESP is.
Is the only test of this functionality currently the "split" test?
@cwsmith W.R.T. testing of the geometric kernels are there any canonical tests that should be performed?
@tuckerbabcock do you have instructions for how to install/run the other necessary components? Does EGADS always require ESP? Unless I had read the comments on this thread I would have no idea what ESP is.
I don't have any specific instructions, other than what's in the readme for how to build ESP: https://acdl.mit.edu/ESP/. ESP (The Engineering Sketch Pad) contains EGADS among other components. One would typically use the ESP interface to build an EGADS model. EGADS is to parasolid what ESP is to NX.
Is the only test of this functionality currently the "split" test?
@cwsmith W.R.T. testing of the geometric kernels are there any canonical tests that should be performed?
I also added a "verify" test, but you're right, there should be more. @cwsmith please let me know if there is some list of canonical tests I should add.