Issue refreshing Wix OAuth tokens - cannot send payload as JSON using `refresh_token_request` compliance hook

[skray]

The crux of my question below is: Am I correct that there is currently no way to use the refresh_token_request to send a JSON payload to the token request endpoint?

To explain the issue I am having, Wix's OAuth flow requires access token requests and refreshes to be made using (non-standard) JSON request bodies, instead of the standard x-www-urlformencoded body. Documentation for reference:

• Access token request: https://dev.wix.com/docs/rest/app-management/oauth-2/request-an-access-token
• Access token refresh: https://dev.wix.com/docs/rest/app-management/oauth-2/refresh-an-access-token

I am able to get the initial access token using the access_token_request compliance hook, using a hook very similar to the one described in the PR that introduced the it:

def redacted_company_compliance_fix(session):
    def _request_in_json_not_form_urlencoded(url, headers, request_kwargs):
        headers["Content-Type"] = "application/json"

        if "redirect_uri" in request_kwargs["data"]:
            del request_kwargs["data"]["redirect_uri"]

        request_kwargs["json"] = request_kwargs["data"]

        del request_kwargs["data"]

        return url, headers, request_kwargs

    session.register_compliance_hook("access_token_request", _request_in_json_not_form_urlencoded)
    return session

However, I am not able to figure out a way to use the refresh_token_request compliance hook in a similar way to send a JSON payload. The refresh logic is much more tightly coupled to sending the request payload as x-www-form-urlencoded, sending only explicit kwargs into the self.post method, including an expectation that the body returned from the refresh_token_request hook is a urlencoded string. Here is the code in question:

for hook in self.compliance_hook["refresh_token_request"]:
    log.debug("Invoking refresh_token_request hook %s.", hook)
    token_url, headers, body = hook(token_url, headers, body)

r = self.post(
    token_url,
    data=dict(urldecode(body)),
    auth=auth,
    timeout=timeout,
    headers=headers,
    verify=verify,
    withhold_token=True,
    proxies=proxies,
)

So:

• Am I correct that there is no way to manipulate the request here to have it send JSON instead of an x-www-form-urlencoded body?
• If so, I am happy to put up a PR, but could use guidance on how to update the refresh_token method and refresh_token_request hook to allow this in a non-breaking way. Would something like this be a good start, allowing non-x-www-form-urlencoded strings to be returned from the refresh_token_request hook?

for hook in self.compliance_hook["refresh_token_request"]:
    log.debug("Invoking refresh_token_request hook %s.", hook)
    token_url, headers, body = hook(token_url, headers, body)

try:
  # attempt to turn body into dictionary
  body = dict(urldecode(body))
except ValueError:
  # allow non-x-www-form-urlencoded bodies to be passed as strings
  pass 

r = self.post(
    token_url,
    data=body,
    auth=auth,
    timeout=timeout,
    headers=headers,
    verify=verify,
    withhold_token=True,
    proxies=proxies,
)

[skray]

I went ahead and opened a PR with my above suggestion just to prove it out as a possibility, but I'm happy to take this any direction needed based on feedback.

[zach-flaglerhealth]

@skray I see I'm not the only one that ran into something like this!

TL;DR: I was misled by the POST on a sandbox web page regarding scopes and urlencoded data vs. json in the body. Leaving this here for posterity.

For the API I'm testing against, they have a sandbox instance, and allow you to get an auth token there to test against the APIs. Looking at that request in the browser, it took the list of scopes requested and concatenated them with '+'s. so, requesting two different scopes, scopeType_A/scopeBranch_A.scope_A and scopeType_B/scopeBranch_B.scope_B would send the request in the body like so:

{
	"grant_type": "client_credentials",
	"scope": "scopeType_A/scopeBranch_A.scope_A+scopeType_B/scopeBranch_B.scope_B"
}

And the header includes Content-Type: application/json. This returned a valid token.

However, when sending it via the requests_oauthlib library, the body was url encoded as:

grant_type=client_credentials&scope=scopeType_A%2FscopeBranch_A.scope_A%2BscopeType_B%2FscopeBranch_B.scope_B

and the API I was working against was returning a 401 that the scope was invalid.

While digging into it, I thought it was simply that the urlencode-ing of the data was what was going wrong, but it turns out that when its sent in this form, it expects spaces instead of plusses.

While digging into this, I found that the code path for parameters ends up in common.py in oauthlib, line 233, which automatically encodes the parameters. Trying to figure out if this was requested before, I found an issue on the oauthlib project asking about the same thing, with the project re-filing it under the requests_oauthlib lib. Lukasa responded that the API is mandated to support application/x-www-form-urlencoded by the OAuth2 specification.

Most of that is not terribly relevant to this exact issue, but I wanted to put this somewhere that it might come up in Google searches, and you had a detailed-enough issue that it helped me track this down.

[skray]

Thanks @zach-flaglerhealth.

To clarify for this issue with Wix, they are very explicitly rejecting x-www-form-urlencoded bodies with an HTTP 415 response code and a body of

{
    "errorCode":409,
    "errorDescription":"[business][RECOVERABLE][Hoopoe Core] o.s.w.HttpMediaTypeNotSupportedException - Content type 'application/x-www-form-urlencoded;charset=UTF-8' not supported",
    "success":false,
    "payload":null
}