Unable to create new session via selenium grid 4 without header charset=utf-8

[andriiTarasov]

Appium or Appium Desktop or Appium Inspector?

• [ ✔️ ] I have verified that my issue does not occur with Appium and should be investigated as an Appium Inspector issue

The problem

I tried to create new appium session via selenium grid 4 using Appium Inspector and have response error - "Failed to create session. Content-Type header does not indicate utf-8 encoded json: application/json".

4444 - port for selenium grid 4.

cURL data from Appium Inspector request - curl 'http://127.0.0.1:4444/session'
-H 'Connection: keep-alive'
-H 'accept: application/json'
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) appium-inspector/2022.1.2 Chrome/91.0.4472.164 Electron/13.6.3 Safari/537.36'
-H 'content-type: application/json'
-H 'Sec-Fetch-Site: cross-site'
-H 'Sec-Fetch-Mode: cors'
-H 'Sec-Fetch-Dest: empty'
-H 'Accept-Language: ru'
--data-raw '{"capabilities":{"alwaysMatch":{CAPS_DATA},"firstMatch":[{}]},"desiredCapabilities":{desiredCapabilitiesData}}'
--compressed

Request response -

{ "value": { "error": "unknown error", "message": "Content-Type header does not indicate utf-8 encoded json: application\u002fjson", "stacktrace": "" } }

Same request sended via cURL with changed Content type: application/json;charset=utf-8 was successful.

Environment

• Appium: v2.0.0-beta.25
• Selenium Grid: 4.1.1 (revision e8fcc2cecf)
• I am running Appium Inspector version 2022.1.2
• I am on (pick one):
  • [ ✔️] Mac
  • [ ] Windows
  • [ ] Linux

[jlipps]

I doubt that the code for sending headers is in the inspector. Rather it would be in the webdriver client used, which is web2driver, built on the webdriverio core code. So that would be where the issue is. Can webdriverio start a session on se grid 4?

[broetchenrackete36]

Same problem with Grid 4.4.0 and Inspector 2022.8.1. Web2Driver 3.0.4 sends the correct header, so the problem seems to be somewhere else?

[mykola-mokhnach]

@christian-bromann Does wdio send the encoding?

[christian-bromann]

Unless defined differently by the user WebdriverIO sends this default header:

'Content-Type': 'application/json; charset=utf-8',

[jlipps]

Can you look in the network panel of the dev tools to see what headers are sent by the inspector?

[broetchenrackete36]

POST /session HTTP/1.1
Host: gridserver:4444
Connection: keep-alive
Content-Length: 756
accept: application/json
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) appium-inspector/2022.8.1 Chrome/91.0.4472.164 Electron/13.6.9 Safari/537.36
content-type: application/json
Accept-Encoding: gzip, deflate
Accept-Language: de

[jlipps]

just curious, what if you try to start a session with web2driver on its own (not the inspector)?

[jlipps]

I just double checked, and there's no code in the inspector that sets content-type. so it'd either be an issue with web2driver or ... something we can't change about how browser APIs work?

[broetchenrackete36]

It seems to me that Electron is the culprit here... Tried a standalone nodejs-app and Web2Driver worked fine (correct header), tried Web2Driver in Electron and the header is without the charset... No idea how to change that though.... See also this pull request: #291 from @KazuCocoa

[broetchenrackete36]

It looks like it should be fixable with the following, but I couldn't get it to work with the inspector, only with my test electron app:

  browserWindow.webContents.session.webRequest.onBeforeSendHeaders(function(details, callback) {
    const requestHeaders = {...details.requestHeaders, ...{
        ['content-type']: 'application/json; charset=utf-8'
    }};
    callback({cancel: false, requestHeaders})
  });

[jlipps]

yikes, that's strange that electron is doing something to override headers! i guess we'll need to play around with something like this until it works. i think we'd also want to just add ; charset=utf-8 to an existing content-type header if one already exists.

[KazuCocoa]

With:

    const serverOpts = {
      hostname: host,
      port: parseInt(port, 10),
      protocol: https ? 'https' : 'http',
      path,
      connectionRetryCount: CONN_RETRIES,
      connectionRetryTimeout: CONN_TIMEOUT,
      headers: {
        'content-type': HEADERS_CONTENT,
        'x-kazu-content-type': HEADERS_CONTENT
      }

in this repository passed to https://github.com/webdriverio/webdriverio/blob/6bbe1cf371f06e517ac62a4e02724c8b610803cb/packages/webdriver/src/request/browser.ts#L52. The options had below:

{
    "method": "POST",
    "retry": 0,
    "followRedirect": true,
    "responseType": "json",
    "throwHttpErrors": false,
    "https": {
        "rejectUnauthorized": true
    },
    "headers": {
        "Content-Type": "application/json; charset=utf-8",
        "Connection": "keep-alive",
        "Accept": "application/json",
        "User-Agent": "webdriver/7.16.14",
        "content-type": "application/json; charset=utf-8",
        "x-kazu-content-type": "application/json; charset=utf-8",
        "Content-Length": "216"
    },
    "timeout": 300000,
    "json": {
        "capabilities": {
            "alwaysMatch": {
                "appium:newCommandTimeout": 3600,
                "appium:connectHardwareKeyboard": true
            },
            "firstMatch": [
                {}
            ]
        },
        "desiredCapabilities": {
            "appium:newCommandTimeout": 3600,
            "appium:connectHardwareKeyboard": true
        }
    },
    "url": "http://localhost:4723/session"
}

but the exact request by the ky lib did not have application/json; charset=utf-8. It only had application/json. Maybe... ky behaved something unexpected in a specific case... x-kazu-content-type" was added expectedly, so the custom headers in webdriverio itself should have worked.

Hm..