feat: Add ExternalDataRepository

[untereiner]

May close #2953

More explanations:

There below is the situation today:

• Every child of MeshGeneratorBase has to fulfill its API even if not compatible
• ExternalMeshGeneratorBase has to have a file as input

classDiagram
namespace GEOS {

  class Group 

 class WellGeneratorABC

 class WellGeneratorBase
  
  class MeshGeneratorBase {
    generateMesh( Group & parent, SpatialPartition & partition ): void 
    importFieldOnArray( Block block, ....) : void virtual
    freeResources(): virtual void 
    getVolumicFieldsMapping(): std::map< string, string > const &
    getSurfacicFieldsMapping(): std::map< string, string > const & 
  }

  class ExternalMeshGeneratorBase {
    m_filePath: Path 
    m_translate: R1Tensor
    m_scale: R1Tensor
    m_volumicFieldsToImport: array1d< string >
    m_volumicFieldsInGEOS: array1d< string >
    m_surfacicFieldsToImport: array1d< string > 
    m_surfacicFieldsInGEOS: array1d< string >
  }

  class InternalMeshGenerator 
}

Group <|-- MeshGeneratorBase
MeshGeneratorBase <|-- ExternalMeshGeneratorBase
MeshGeneratorBase <|-- InternalMeshGenerator
Group <|-- WellGeneratorABC 
WellGeneratorABC <|-- WellGeneratorBase

What I expect after this PR:

• A new class (ExternalDataRepository) with almost no constraints on what it expects allows me to create a proper API for another module (example: RESQML)
• A new class (VTKHierarchicalDataRepository) to add the capability to load complex meshes with semantics (example provided on singlePhaseFlow)
• A new class (Region) to add the capability to load sets of cells (vtkDataSets) from a complex mesh data format (VTK or RESQML) and give it a id for internal purpose (aka regionAttribute for vtu files)

classDiagram
direction LR
namespace GEOS {

class Group

class ExternalDataRepository {
  virtual open(); void
}

class MeshGeneratorBase {
  generateMesh( Group & parent, SpatialPartition & partition ): void 
  importFieldOnArray( Block block, ....) : void virtual
  freeResources(): virtual void 
  getVolumicFieldsMapping(): std::map< string, string > const &
  getSurfacicFieldsMapping(): std::map< string, string > const & 
}

class ExternalMeshGeneratorBase {  
  m_translate: R1Tensor
  m_scale: R1Tensor
  m_volumicFieldsToImport: array1d< string >
  m_volumicFieldsInGEOS: array1d< string >
  m_surfacicFieldsToImport: array1d< string > 
  m_surfacicFieldsInGEOS: array1d< string >
}

class VTKMeshGenerator {
  m_filePath: Path 
}

class InternalMeshGenerator 

class VTKHierarchicalDataRepository

class MeshComponentBase

class WellGeneratorBase

class Region

}

Group <|-- ExternalDataRepository
Group <|-- MeshGeneratorBase
MeshGeneratorBase <|-- ExternalMeshGeneratorBase
MeshGeneratorBase <|-- InternalMeshGenerator
ExternalMeshGeneratorBase <|-- VTKMeshGenerator
ExternalDataRepository <|-- VTKHierarchicalDataRepository
Group <|-- MeshComponentBase
MeshComponentBase <|-- WellGeneratorBase
MeshComponentBase <|-- Region


namespace GEOS-RESQML {

class EnergyMLDataObjectRepository {
common::DataObjectRepository * getData()
COMMON_NS::AbstractObject* getDataObject(string const & id)
COMMON_NS::AbstractObject* getDataObjectByTitle(string const & name)

}


class EpcDocumentRepository

class RESQMLMesh 
}

ExternalDataRepository <|-- EnergyMLDataObjectRepository
EnergyMLDataObjectRepository <|-- EpcDocumentRepository
MeshGeneratorBase <|-- RESQMLMesh

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 34.70588% with 111 lines in your changes missing coverage. Please review.

Project coverage is 57.17%. Comparing base (b23c80e) to head (ebc60b9). Report is 65 commits behind head on develop.

Files with missing lines	Patch %	Lines
...oreComponents/mesh/generators/VTKMeshGenerator.cpp	28.84%	37 Missing :warning:
...ents/mesh/generators/VTKHierarchicalDataSource.cpp	0.00%	28 Missing :warning:
src/coreComponents/mesh/ExternalDataSourceBase.cpp	17.64%	14 Missing :warning:
.../coreComponents/mesh/ExternalDataSourceManager.cpp	33.33%	14 Missing :warning:
src/coreComponents/mesh/generators/Region.cpp	0.00%	12 Missing :warning:
src/coreComponents/mesh/generators/Region.hpp	33.33%	2 Missing :warning:
...ents/mesh/generators/VTKHierarchicalDataSource.hpp	33.33%	2 Missing :warning:
...reComponents/mesh/generators/MeshGeneratorBase.cpp	66.66%	1 Missing :warning:
...oreComponents/mesh/generators/VTKMeshGenerator.hpp	50.00%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2957      +/-   ##
===========================================
- Coverage    57.22%   57.17%   -0.05%     
===========================================
  Files         1143     1149       +6     
  Lines        98913    99044     +131     
===========================================
+ Hits         56600    56628      +28     
- Misses       42313    42416     +103     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[paveltomin]

hi @untereiner i am not sure how adding MeshBase layer helps here, most probably i am missing soemthing could you please add some description explaining that? thank you

[untereiner]

Hi @paveltomin, I can link to the RESQML plugin. Adding this layer helps me creating a child node in Mesh by subclassing something more abstract that MeshGeneratoreBase

The entry point of the RESQML C++ library is a repository of 3D objects that I want to put in the deck.

I can also link an example deck if needed.

Also, VTK has file formats storing composite data (grids and surfaces) with complex relationships ( see here). I claim that these files should be considered as « data repositories » to extract from them what’s needed where it’s needed. So, this MeshBase layer could also benefit to them

[paveltomin]

Hi @paveltomin, I can link to the RESQML plugin. Adding this layer helps me creating a child node in Mesh by subclassing something more abstract that MeshGeneratoreBase

The entry point of the RESQML C++ library is a repository of 3D objects that I want to put in the deck.

I can also link an example deck if needed.

Also, VTK has file formats storing composite data (grids and surfaces) with complex relationships ( see here). I claim that these files should be considered as « data repositories » to extract from them what’s needed where it’s needed. So, this MeshBase layer could also benefit to them

thanks for the explanations!

[MelReyCG]

@untereiner does it modify this graph in the docs ?

[untereiner]

@MelReyCG it adds a level in the class hierarchy, so yes. The PRs of Thomas are modifying these files. I don’t know how to move this forward.

[untereiner]

Hi @rrsettgast, I put diagrams to explain a bit more what this PR is expected to improve

[untereiner]

Hi @untereiner . Would it be better to rename the proposed MeshBase to MeshGeneratorBase, and rename the current MeshGeneratorBase to something more specific to its use case?

I am open to suggestions ! I don’t have any good idea since MeshGeneratorBase is really ment to be a parent of a specific family of generators.

[untereiner]

@rrsettgast I completely changed the design. This one is closer and cleaner to what the architecture really is. Think of EpcDocumentRepository as a database with an API you query to get what's inside (meshes, properties, geometric objects, etc.)
Here is an excerpt of a simulation deck:

<Problem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <Solvers>
   ....
  </Solvers>

  <ExternalDataRepository>
    <EpcDocumentRepository
      logLevel="2"
      name="plop"
      files="{ myfile.epc }"/>
  </ExternalDataRepository> 
  

  <Mesh>
    
    <RESQMLMesh
      logLevel="2"
      name="mesh"
      repositoryName="plop"
      uuid="5cff7c6a-b5d7-441f-b981-bc77e91a144d">

      <Region
        name="Region 1"
        uuid="0b807d31-540a-45e7-a898-b75b05e358a4"        
      />
      <Region
        name="Region 2"
        uuid="1facf29e-25ec-4c6c-b404-595dc84eb29c"
      />
      <Region
        name="Region 3"
        uuid="4b3ce11e-6b9a-4497-b430-d06226a0491c"
      />
      <Region
        name="Region 4"
        uuid="31e184e2-7894-4408-bb57-d0ec10f1dcbf"
      />
      <Region
        name="Region 5"
        uuid="4efccb61-13f4-4583-8f3f-8303be28ea88"
      />

      <Property
        name="rockPermeability_permeability"
        uuid="c2f3faae-4a88-4906-95ff-01e2bbc95c5f"
        title="PERM"
      />
      <Property
        name="rockPorosity_referencePorosity"
        uuid="f678acb4-eecf-4dd9-8a02-2cd7a64179eb"
        title="PHIT"
      />
      <Property
        name="netToGross"
        uuid="c27a314a-2ead-4ed6-8e79-4087cf1557bd"
        title="NTG"
      />

      <Surface uuid="" title=""/>
      <Surface uuid="" title=""/>

    </RESQMLMesh>

  </Mesh>
   ...
</Problem>

I think this kind queryable external data repository will also be usefull with vtkParitionedDataSetCollections because you have to use a DataAssembly to query the hierarchical organization of partitioned-datasets.

[rrsettgast]

@untereiner Can we discuss this at the meeting tomorrow?

[untereiner]

@rrsettgast I am not available at the meeting time. Can we meet at another time or next week ?

[untereiner]

Here is an insight for VTK complex datasets

<Problem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

  <Solvers>
   ....
  </Solvers>

  <ExternalDataRepository>
    <VTKHierarchicalDataRepository
      logLevel="2"
      name="plop"
      files="{ myfile.vtpc }"/>
  </ExternalDataRepository> 
  

  <Mesh>
    
    <VTKMesh
      logLevel="2"
      name="mesh"
      repositoryName="plop"
      path="/Root/grid">
     
     <InternalWell
        logLevel="1"	
        name="wellInjector"
        wellRegionName="well"
        wellControlsName="wellControls"
        repositoryName="plop"
        path="/Root/wells/wellInjector">
         
        <Perforation
           name="wellInjector_perf0"
           distanceFromHead="5"/>

     </InternalWell>

    </VTKMesh>

  </Mesh>
...
</Problem>

[rrsettgast]

@rrsettgast I am not available at the meeting time. Can we meet at another time or next week ?

Hello @untereiner. Next week is fine. Thanks

[untereiner]

ExternalDataRepository is too close to the data repository name used internally by GEOS: Here are some suggestions Choose a comment emoji to cast your vote

• :heart: ExternalDataSource
• :rocket: : ExternalDataStream
• :tada: : ExternalDataInput

[untereiner]

@rrsettgast I will update the branch and change for ExternalDataSource then I should be ok for merging

[wrtobin]

Nothing jumps out at me in the implementation that needs changes, really just the naming.

[paveltomin]

It seems like we still need to rename ExternalDataRepository > ExternalDataSource

i renamed it, could you please have another look?

[paveltomin]

@wrtobin could you please have another look?