KhiopsML
Enhancement: ignore scenario template sections for which JSON value is missing
Description
Currently the search-replace engine expects all values present in scenario templates to be also present in the JSON provided to replace template placeholder. This is a proposal to permit missing JSON values. When such missing value will be encountered, the search/engine will drop the corresponding scenario template fragment.
Questions/Ideas
The origin of this proposal was to simplify handling of optional values, for example memoryLimit or maxProcNumber, but it could be generalized to all scenario template values. The missing values would simply be ignored and corresponding scenario sections would be dropped. More specifically:
- for single line with a missing value -> drop line
- for a conditional section with missing conditional value -> drop conditional block
- for a loop section with missing array value -> drop loop section
In order to ease scenario template debugging and find template values never used (and hence possibly not properly maintained anymore) we could add a debugging output to the khiops -O operation. With this mode active, khiops would output a warning for any template parameter without corresponding value. This mode can then be used in dedicated tests that use a reference implementation (khiops-python) to generate a fully defined JSON parameter set and detect possible discrepancies between the template and reference implementation.
Context
We are designing a cross-language API generation framework for Khiops. At the start of the journey we extract an abstract representation of the API from khiops-python, being the reference implementation. This representation contains - for each operation - the parameters and corresponding scenario template which will then be filled with parameters and executed by khiops to implement the chosen operation. The parameters fall into the following categories:
- mandatory: must be provided
- optional with default: can be omitted, default will apply (will be injected in scenario, or omitted like below)
- optional without default: can be omitted, if absent no output should be written in scenario template so that khiops can implement a sensible default
- extra runtime parameter: will not go into scenario, should be passed as khiops command line flags
Generic API implementation can be decomposed as follows:
- for a given operation, receive parameter set in protobuf format (by default in proto, any value set to default or empty arrays are automatically removed to produce more compact encoding)
- convert parameter set in JSON format
- prior to this proposal, we add all default values to the message to satisfy scenario template inputs, including empty arrays
- with this proposal, the JSON conversion is equivalent to the proto, any redundant value (default or empty array) is cleaned
- extra runtime parameters are extracted (they will be used as command line parameters) and resulting JSON is passed to khiops -j with the appropriate scenario template for the operation
- following execution, analyze return code and logs to give informative feeback to the user.
Précision: ceci n'est pas encore une spécification. Il s'agit essentiellement de réflexions en cours, à maturer, en vue de finaliser la refonte du moteur de templating pour python plus une API protobuf.
Suite à un Poc d'extraction automatique de l'API par introspection des fonctions de l'API core, via leur signature et leur docstring, cette proposition parait pertinente.
Je propose même de la généraliser en autorisant les valeurs null dans les json et en les traitant comme des valeur manquantes.
En effet, les locals d'une fonction python sont stockés sous la forme d'un dictionnaire ayant autant de clés que d'argument dans la fonction, avec une valeur None associée aux argument dont la valeur par défaut est None, qui se traduit par la valeur null une fois exporté dans un fichier json.