Document arff-trace for parameter sweeps

[mfeurer]

Currently, there is no documentation of how the optimization trace of a parameter sweep is stored. To the best of my knowledge, this file describes it:

http://www.openml.org/data/download/2035727/optimization_trace6946920946676621819.arff

And some documentation, pasted from an email from @janvanrijn

The following fields are mandatory: repeat, fold, iteration, evaluation and selected. Furthermore, all fields that involve an optimized parameter are indicated with the prefix "parameter_" (in line with my email from May 24). All other fields are facultative.

[mfeurer]

Also, it would be great to have documented on how to save the runtime of individual models, as would be interesting in other contexts (such as https://github.com/openml/openml-python/pull/1038#discussion_r607917705)

[janvanrijn]

Hi Matthias,

Thanks for reminding me of this. The current documentation is far from perfect, but I am happy to put in an effort to improve it.

• I found this old document, that consists of a description of the run uploads: https://github.com/openml/OpenML/wiki/Run-Uploading (note that this document is no longer visible to users)
• Additionally, this is what is now documented (it misses some information, and got some information wrong from the top of my head): https://docs.openml.org/OpenML_definition/ -> Run -> Trace

My proposal would be the following:

• I can integrate the information from the currently hidden document into the currently public document
• I can add a link to an example trace document

Any other suggestions?

[mfeurer]

That sounds like a good proposal.

[joaquinvanschoren]

Hi all,

Did you see the API docs? https://new.openml.org/api Under run: /run/trace/{id} I do notice now that the example has a broken link.

More documentation is always better, but I would at least cross-link them?

[janvanrijn]

Hi Joaquin,

Thanks for this new documentation, but there seems to be a broken link: https://new.openml.org/api/v1/xml_example/run.trace

I think this documentation also doesn't describe the arff file that needs to be uploaded. I will add a short description at the docs.openml.org site, presuming that this documentation will also be available on the new website?

Best, Jan

Best, Jan

Op za 10 apr. 2021 om 00:16 schreef Joaquin Vanschoren < @.***>:

Hi all,

Did you see the API docs? https://new.openml.org/api Under run: /run/trace/{id} I do notice now that the example has a broken link.

More documentation is always better, but I would at least cross-link them?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openml/OpenML/issues/413#issuecomment-817000561, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARPX6QALDOJIFUGY7MFRRLTH54EZANCNFSM4DKDHNZA .

[janvanrijn]

I created two PR's, the one above (which should be evaluated first), and then one on the documentation repo (which could be reviewed after)

Happy to hear your suggestions

[mfeurer]

Hey, I just checked https://github.com/openml/docs/pull/15/files and the links aren't working yet. Does there need to be a release of the server for them to appear?

[mfeurer]

The documentation PR https://github.com/openml/docs/pull/15 just got merged. @joaquinvanschoren who can update the new API docs to add a link to the new description?