[Feature Request] required_content for get_notes()

[carschno]

This is probably a feature request.

I am trying to get the acceptance status for a number of notes (submissions). I am roughly following the example from the documentation, but on a per-note basis. Another difference to the example is that I would like to retrieve the status, rather than just accepted papers.

So this is my current code:

    decision_notes = [
        note
        for note in client.get_notes(forum=note.forum)
        if "decision" in note.content
    ]

    if decision_notes:
        acceptance_status = decision_notes[0].content["decision"]

This works fine, but seems very slow. For a large number of notes, I would prefer to have something like the content parameter in get_notes() which allows to filter for the presence of a specific field, ignoring the value. For instance a required_content parameter so I could do this:

    decision_notes = [
        note
        for note in client.get_notes(forum=note.forum, required_content="decision")
    ]

This should save a lot of network traffic because the filtering could be done on the server side. Not sure if this makes sense, and whether it is currently possible though.

[melisabok]

If you want to get all the decision notes no matter the value of the decision, you can get all the notes by invitation, like the example says:

all_decision_notes = openreview.tools.iterget_notes(client, invitation = 'MIDL.io/2019/Conference/-/Paper.*/Decision')

if you want to get the decision for a specific note:

client.get_notes(forum=note.forum, invitation='MIDL.io/2019/Conference/-/Paper.*/Decision')

or

client.get_notes(forum=note.forum, invitation='MIDL.io/2019/Conference/-/Paper' + str(note.number) + '/Decision')

[carschno]

@melisabok Great, thanks a lot!

I suppose I can use a more generous wildcard as the forum should be unique, such as:

client.get_notes(forum=note.forum, invitation='.*/Decision')

Is that a correct assumption? This way, I do not need to track back which invitation a note came from initially.

Update: trailing string seems not to be consistent at all. So far, I have seen Decision, acceptance, and sometimes the acceptance is part of the submission note.

Related to that, could you tell me more about the exact regular expressions subset that openreview.net processes?

This gives me the right result:

client.get_notes(forum="rJV7l2VFg", invitation='.*/acceptance')

Whereas this does not:

client.get_notes(forum="rJV7l2VFg", invitation='ICLR.cc/2017/workshop/-/paper99/(acceptance|Decision)')

[melisabok]

Using a .* at beginning of the filter can be too expensive because an index is not being used to do the search. We prefer to get the decision by venue. Not all the venues have the same invitation name, this convention has changed over the years, for example: 'acceptance' -> 'decision'.

And there are also venues that don't have this decision note and the decision is stored in the Meta_Review invitation, for example: ICLR 2019.

My advice is to pick one note of each venue and call client.get_notes(forum=[forum_id]). Then inspect the result and check all the available invitations.

We are in the process of standardize the decision and save it in the note itself. We will let you know when it is ready.