Allow POST requests

[giovannipizzi]

Now the requests can only be GET. In #21 we are removing the /all endpoint that was the only place mentioning POST requests. We should probably say that POST requests are allowed everywhere since query strings can become very long for non-trivial filters (also mentioning @rartino who I think is supportive)

[giovannipizzi]

I'm appending here a sentence from the text removed in #21 that can be adapted and reused:

The general entry listing endpoint MUST accept both GET and a 
POST-type requests, with provided POST-type URL query parameters 
overriding duplicate URL query parameters provided as GET URL 
query parameters.

[giovannipizzi]

Should we add this to the 1.0 milestone?

[blokhin]

Are you really sure the POST is needed? Two observations on that:

• according to the HTTP semantics, POST modifies the content, and apparently OPTiMaDe is far from that at the moment

• length considerations: in HTTP specs GET has no length limit defined. Some web-servers have a default limit of 8KB, in practice reportedly it's safe to use up to 2000 chars. Anyway whatever complex GET-request would fit (please, correct me if I'm wrong)

[merkys]

While I'm not in favor of allowing POST per se (as @blokhin remarked, POST is reserved for database modifications), there are various restrictions on the length of GET request string. 2000 chars might seem a lot, but complex queries on nested property names easily approach it (to illustrate, a "simple" example from the specs is already 260 chars long).

[rartino]

Is there a reference on that POST "modifies the content" / "is for database modifications"? Are you mixing up POST and PUT?

The defining sentence of POST in RFC 7231 is "The POST method requests that the target resource process the representation enclosed in the request according to the resource's own specific semantics." It goes on to list examples, some of which modifies data, but the very first example is "Providing a block of data, such as the fields entered into an HTML form, to a data-handling process;" Isn't what we are doing more or less exactly providing a "block of data" (including, e.g., a massive filter) to a "data-handling process"?

This is in difference to PUT, which says: "The PUT method requests that the state of the target resource be created or replaced with the state defined by the representation enclosed in the request message payload."

In my opinion we should just say that the API implementation MUST support that any URL query parameter may instead be provided via POST. I don't see why it would be difficult to support in implementations, and it avoids arbitrary length limitations.

[merkys]

Oh, my perception of POST/PUT was formed by now-obsolete RFC 2616 (obsoleted by RFC 7231, mentioned by @rartino).

In the light of this, I agree that we may allow using GET and POST interchangeably.

[sauliusg]

I'm appending here a sentence from the text removed in #21 that can be adapted and reused:

The general entry listing endpoint MUST accept both GET and a 
POST-type requests, with provided POST-type URL query parameters 
overriding duplicate URL query parameters provided as GET URL 
query parameters.

I would be definitely against "MUST" since it puts extra (duplicate!) effort on implementer. Let's not succumb to creeping featurism!

[sauliusg]

I'm appending here a sentence from the text removed in #21 that can be adapted and reused:

The general entry listing endpoint MUST accept both GET and a 
POST-type requests, with provided POST-type URL query parameters 
overriding duplicate URL query parameters provided as GET URL 
query parameters.

My understating of the REST is that POST is usually supposed to modify the resource (see RFC 7321, section "POST"). Now this does not, as I understad, forbid using POST for queries, just would take up a function point...

[rartino]

@sauliusg Your point about POST vs GET was discussed upthread. The RFC you link lists example uses of POST and one of those examples is similar to ours.

If we don't put MUST we can just as well not mention this in the specification. I don't see anything in the specification right now that prevents me from serving POST if I want to.

The extra work to serve identically formed GET and POST requests is truly minimal in all web frameworks I've used. The only thing one needs to do is to stuff the POST fields into the same data structure that one put the GET url parameters in.

Isn't it nice to have a standardized way to submit arbitrarily-length filter strings?

[sauliusg]

Isn't it nice to have a standardized way to submit arbitrarily-length filter strings?

Yes, this would be a very good feature.

[sauliusg]

If we don't put MUST we can just as well not mention this in the specification. I don't see anything in the specification right now that prevents me from serving POST if I want to.

Yes, we should not put a MUST. Probably MAY.

[sauliusg]

Your point about POST vs GET was discussed upthread. The RFC you link lists example uses of POST and one of those examples is similar to ours.

Sorry, I've overloked the reference but yes, its essentially the same how understand the situation.

So in a few words:

• we can specify the use POST for sending queries;

• POST would circumvent the existing QS length limitations;

• We can send filter strings using GET and POST equally well;

• We can encode queries in any other formats (but of course the bloated ones like JSON will be little suited for GET+QS); but whether we want/need/can suppot them is a separate orthogonal issue (#267)

[giovannipizzi]

My opinion: I'm supportive of allowing both POST and GET, and for now use the same syntax between the two, whatever the syntax is (so both for the client and the server, it's just a matter of deciding which method to use)

[merkys]

If we don't put MUST we can just as well not mention this in the specification. I don't see anything in the specification right now that prevents me from serving POST if I want to.

Yes, we should not put a MUST. Probably MAY.

If we decide to make POST support OPTIONAL, it becomes a feature a client cannot rely upon. I mean if I was to implement an "universal" OPTIMADE client (or anything skyscanner-like), I would not use POST as only a subset of providers support it. Instead, I would still use GET and would have to cater to its possible shortcomings. In short, I believe that all OPTIONAL convenience features are actually not that convenient.

[fekad]

My opinion: I'm supportive of allowing both POST and GET, and for now use the same syntax between the two, whatever the syntax is (so both for the client and the server, it's just a matter of deciding which method to use)

Could I ask which content type would be preferable for the POST request? usage example

1. The default application/x-www-form-urlencoded where you can use the same string that you would use for the GET requests, or
2. the very common application/json format which we also use heavily for the OPTIMADE API (of course for the fields you can use the same syntax), or
3. both or,
4. basically any of them including support for eg. XML?

[rartino]

With application/x-www-form-urlencoded we would retain the existing protocol completely, with just your web framework delivering parameters in a POST dictionary instead of a GET dictionary, so I think it makes the most sense to use as the primary supported format.

Support for other submission formats can of course be discussed, just as we have support for other custom response formats. In fact, I think it would be great to put in the specification that you are allowed to accept things like application/vnd.optimade-_exmpl_+json as your provider-specific json-based format and application/vnd.optimade-_exmpl_+xml, etc. If a specific format becomes popular, we can then standardize it to be the meaning of, e.g., application/json.

We need to register application/vnd.optimade, though. Perhaps we should do that?

[merkys]

Could I ask which content type would be preferable for the POST request? usage example

The following are widely used, and easily supported:

• application/x-www-form-urlencoded
• multipart/form-data

We need to register application/vnd.optimade, though. Perhaps we should do that?

I don't see the need to do that. We can reuse already existing carrier formats, and I can't think of any feature in OPTIMADE that would push us to invent OPTIMADE-specific request format.

[rartino]

 We need to register application/vnd.optimade, though. Perhaps we should do that?

I don't see the need to do that. We can reuse already existing carrier formats, and I can't think of any feature in OPTIMADE that would push us to invent OPTIMADE-specific request format.

Did you not understand my examples? How do you propose to handle provider-specific POST formats without risking collision with formats that we standardize at a later point?

[sauliusg]

With application/x-www-form-urlencoded we would retain the existing protocol completely, with just your web framework delivering parameters in a POST dictionary instead of a GET dictionary, so I think it makes the most sense to use as the primary supported format.

I would be in favour of application/x-www-form-urlencoded for exactly the same reason.

[merkys]

I don't see the need to do that. We can reuse already existing carrier formats, and I can't think of any feature in OPTIMADE that would push us to invent OPTIMADE-specific request format.

Did you not understand my examples? How do you propose to handle provider-specific POST formats without risking collision with formats that we standardize at a later point?

As said, I can't think of any reason of inventing application/vnd.optimade POST format. But maybe I'm just shortsighted.

[rartino]

@merkys

As said, I can't think of any reason of inventing application/vnd.optimade POST format. But maybe I'm just shortsighted.

Let me try to spell out my example as clearly as possible.

First, look at the similar situation with the response_format parameter. We agreed that if anyone invents their own response format, they should prefix it with their database prefix. The reason is that: Let's say I invent my own HDF5 format and make it available as response_format=hdf5. What if OPTIMADE later standardizes an official HDF5 format that is subtly different? Then users of my database will be confused over what they get with response_format=hdf5. So, I am instead supposed to call my own version of the format response_format=_exmpl_hdf5 so we avoid this possible future collision.

Now, onto the case with submission formats.

@fekad above suggests he wants to do POST with JSON-encoded data. Let's say he implements his own JSON representation of the OPTIMADE submission data and applies it to POSTs done as application/json. What if OPTIMADE later standardizes on a subtly differently organized JSON format for the submissions of type application/json? Now users will be similarly confused over which variant of these JSON formats they are supposed to POST to @fekad's database. So, we want him to name his own JSON POST format differently, e.g., "_exmpl_json".

However, one is not allowed to just make up media formats, so he really should not use something like application/_exmpl_json. So, how should @fekad represent this database-specific JSON POST format? My suggestion is that if we register application/vnd.optimade, he could do something like application/vnd.optimade-_exmpl_+json (or maybe using some other degree of freedom allowed for custom media formats, e.g., media type parameters)

Does this make it more clear what use an application/vnd.optimade media type could have?