run/evaluate API call is missing

[joaquinvanschoren]

Description

There doesn't seem to exist a Python binding for the /run/evaluate call of the OpenML REST API (see details in https://new.openml.org/api). This call is needed to allow an external evaluation engine to submit new evaluations. This is critical as we are moving away from the older Java evaluation engine.

Expected Results

A run_evaluate method in openml.runs

Actual Results

There is none.

Versions

N/A

[PGijsbers]

@joaquinvanschoren The XML schema reads:

<xs:element name="evaluation_engine_id" type="xs:integer"/>
<!--
 The integer id of the evaluation engine (see table).  
-->

which table is being referred to?

And

<xs:element name="evaluation" minOccurs="0" maxOccurs="unbounded" type="oml:evaluation"/>
<!--
 all data that served as input for the run. Determined server side, will be ignored on uploading 
-->

Seems that the documentation at least suggested provided evaluation data is not supported?

Finally, including any form the evaluation's sample attribute, I get the error returned code 434: Illegal combination of evaluation measure attributes (repeat, fold, sample) - Measure(s): fakeresult(0, 0, 0). I assume that this may be caused because the task is not a learning curve task? When I don't provide the sample, I get code 428: Database insertion error, probably due to high server load. - None. I am configured to use the live environment because the test server is not currently running, though I am only trying to upload evaluations for a run I own (that I will delete afterwards).

<oml:run_evaluation xmlns:oml="http://openml.org/openml">
    <oml:run_id>10587713</oml:run_id>
    <oml:evaluation_engine_id>0</oml:evaluation_engine_id> <!-- not sure what a correct id would be -->
    <oml:evaluation repeat="0" fold="0">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="1">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="2">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="3">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="4">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="5">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="6">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="7">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="8">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
    <oml:evaluation repeat="0" fold="9">
        <oml:name>fakeresult</oml:name> <!-- metric name -->
        <oml:value>1</oml:value>
    </oml:evaluation>
</oml:run_evaluation>
"""