ArtifactGenerationExtensions

[MauricioUyaguari]

Related PRS

• [x] Add definition and dataspace implementation - https://github.com/finos/legend-engine/pull/833
• [x] Update definition based on 8/31 discussion - https://github.com/finos/legend-engine/pull/900
• [x] Add handling to sdlc pipeline - https://github.com/finos/legend-sdlc/pull/481
• [ ] Add reading and storing artifacts to depot https://github.com/finos/legend-depot/pull/98
• [ ] legend-studio use depot api to fetch dataspace analytics

Description of Problem

We want to generate useful artifacts for specific Packageable Elements. This is useful for caching the contents of these artifacts through legend-depot to improve performance when fetching these artifacts.

This comes in handy when doing complicated analytics on Packageable elements which takes time and resources to recalculate. An example is generating an ExecutionPlan for a service element. With this feature, the generation of the execution plan will occur once during the build step of the project and can therefore be used whenever the generation plan for the service is needed (i.e executing query etc.).

Solution Overview

We already have a maven plugin to generate files for an sdlc project. Right now it reads FileGeneration elements defined in the GenerationSpecification. The limitation here is that the user must specify the generation and also it is tied to having a generator driven by a schema/code generation. We will define the ArtifactGenerationExtension more loosely. The main driver will be a packageable element, which will now be referred as the generator, in the graph that can define its own generation.

Current Approach: Use element path as driver for file paths

ArtifactGenerationExtension Diagram

File Structure

The generate method of the ArtifactGenerationExtension will return a list of artifacts with filePaths on them. When seralizing them to files we will place these files under the sub folder with the name tied to the generator that generated these files + the extension root folder.

Example

Generators with files generated

• model::CodeGeneration : (extension with root path code-generation)
  • generated/MyJava.java
  • MyPython.py
• model::CodeGeneration2 : (extension with root path code-generation)
  • MyPthon.py
• model::SchemaGeneration
  • schemaGen/MyJson.json : (extension with root path schmea-generation)
  • MyProto.proto
• model::MyClass : (extension with root path other-generation)
  • someGeneration/stuff.txt
  • MyStuff.txt

File Structure on file generation jar output

• model
  • CodeGeneration
    • code-generation
      • generated
        • MyJava.java
      • MyPyton.py
  • CodeGeneration2
    • code-generation
      • MyPyton.py
  • SchemaGeneration
    • schema-generation
      • schemaGen
        • MyJson.json
      • MyProto.proto
  • ClassGeneration
    • other-generation
      • someGeneration
        • stuff.txt
      • MyStuff.txt

Impact on Depot

• Currently depot uses classifier path to index files generated through the file generation maven plugin by:
  • fetching GenerationSpecification entity through classifier path.
  • fetching fileGenerations defined in GenerationSpecification
  • for each fileGeneration we use either the generationOutputPath if defined in the fileGeneration path or we use the file generation path to determine the path to index
  • this path will be indexed as the location of where the expected files generated by this fileGeneration will be located.
• Enhancement We should update apis to unpack all files generated as part of file generation maven plugin. We con potentially query files generated by generator.

Abandoned Approach: Use of RootPaths per extensions

• abandoned on 08/31 due to
  • its effect on depot causing chaos on file distribution and lineage between the generator (packageable elements) and the path location of the files.
  • Extension changes cause non backward compatible changes for depot and clients using depot

Approach (Abandoned )

We will define the ArtifactGenerationExtension more loosely. It is driven by a packageable element in the graph that can define its own generation.

The generate method on the extension will be the main driver of the extension and will expect the element with its compiled model PureModel to produce artifacts.

Impact on Depot (Abandoned )

Current depot apis rely on the paths of the files generated. This will not change for the current FileGeneration element as the location of the files generated by those elements will stay the same and will continue to be defined (for now in GenerationSpecification)

Folder Structure (Abandoned )

All files generated by ArtifactGenerationExtension will be generated under the root folder artifacts-generation-root. Furthermore, each extension will define its artifactsRootPath that will create a folder under the root folder for the files generated by the extension

• (root) artifacts-generation-root
  • (extension1) extenstion1-root-path
    • generatedFile.json (generated by extension1)
  • (extension2) extenstion2-root-path
    • generatedFile.json (generated by extension2)