Getting 202 when latest version of the app is used

[vsingal-p]

Hi

I have deployed mergeable in my own environment, by making a copy of the repo

It was working fine till now (till SHA 5150b7d30e1d6699a92a518e0541a7388b475b09). Today I upgraded the code to latest master (after a month) and things broke.

ERROR (HttpError): If you would like to help us test the Checks API during its preview period, you must specify a custom media type in the 'Accept' header. Please see the docs for full details.
    HttpError: If you would like to help us test the Checks API during its preview period, you must specify a custom media type in the 'Accept' header. Please see the docs for full details.
        at /app/node_modules/@octokit/request/dist-node/index.js:66:23
        at runMicrotasks (<anonymous>)
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async sendRequestWithRetries (/app/node_modules/@octokit/auth-app/dist-node/index.js:445:12)
        at async Job.doExecute (/app/node_modules/bottleneck/light.js:405:18)

The check at the PR is stuck, and is not receiving any response The service is returning a 202

Details of deployment

1. The service is deployed as a web service in an AWS ECS cluster
2. Github app points to the LoadBalancer of this ECS service

Where should I check what's wrong?

[shine2lay]

@vsingal-p the first thing that came to my mind is to re install the node packages, use the correct node and npm version, I think the version you had was before the [email protected] update, that update caused us some issues regarding the environment set up as well.

[vsingal-p]

To clarify (don't have much experience in node based apps) -

1. re install the node packages -> means running npm install or yarn --ignore-engines .. right?
2. use the correct node and npm version -> means updating engines in package.json and updating .nvmrc file
3. Yes, I had pre [email protected] version

[vsingal-p]

1. I updated .nvmrc to use 12.20.1
2. I didn't update engines field in package.json

One question, mergeable uses 14.15.4. Will it work with 12.20.1 if i use one? Currently my deployment environment is not ready for 14.15.4

[vsingal-p]

So I deployed using node 14.15.4 The error I showed above is gone..but it still responds with 202 and the checks on PR are not working

Here's what the execution log looks like

INFO (action/checks): {"logType":"action_before_validate_execute","eventId":"50656700-787d-11eb-90f5-9b0164cf0eed","repo":"pr-mergeable","actionName":"checks","settings":"{\"when\":\"pull_request.*, pull_request_review.*\",\"name\":\"Core approval check\",\"validate\":[{\"do\":\"approvals\",\"min\":{\"count\":1},\"limit\":{\"users\":[\"vsingal\",\"rupawar\"]}}],\"pass\":[{\"do\":\"checks\",\"state\":\"completed\",\"status\":\"success\",\"payload\":{\"title\":\"Mergeable Run has been Completed!\",\"summary\":\"All the validators have returned 'pass'! \\n Here are some stats of the run: \\n {{validationCount}} validations were ran\"}}],\"fail\":[{\"do\":\"checks\",\"state\":\"completed\",\"status\":\"failure\",\"payload\":{\"title\":\"{{failCount}}/{{validationCount}} Fail(s): {{#each failures}} {{toUpperCase name}}{{^@last}}, {{/@last}}{{/each}}\",\"summary\":\"### Status: {{toUpperCase validationStatus}}\\n\\n        Here are some stats of the run:\\n        {{validationCount}} validations were ran.\\n        {{passCount}} PASSED\\n        {{failCount}} FAILED\\n      \",\"text\":\"{{#each validationSuites}}\\n#### {{{statusIcon status}}} Validator: {{toUpperCase name}}\\n{{#each validations }} * {{{statusIcon status}}} ***{{{ description }}}***\\n       Input : {{{details.input}}}\\n       Settings : {{{displaySettings details.settings}}}\\n       {{/each}}\\n{{/each}}\"}}],\"error\":[{\"do\":\"checks\",\"state\":\"completed\",\"status\":\"action_required\",\"payload\":{\"title\":\"Mergeable found some failed checks!\",\"summary\":\"### Status: {{toUpperCase validationStatus}}\\n      Some or All of the validators have returned 'error' status, please check below for details\\n      \\n      Here are some stats of the run: \\n      {{validationCount}} validations were ran. \\n      {{passCount}} ***PASSED***\\n      {{failCount}} ***FAILED***\\n      {{errorCount}} ***ERRORED***\",\"text\":\"{{#each validationSuites}}\\n#### {{{statusIcon status}}} Validator: {{toUpperCase name}}\\nStatus {{toUpperCase status}}\\n{{#each validations }} * {{{statusIcon status}}} ***{{{ description }}}***\\n       Input : {{{details.input}}}\\n       Settings : {{{displaySettings details.settings}}}\\n       {{#if details.error}}\\n       Error : {{{details.error}}}\\n       {{/if}}\\n       {{/each}}\\n{{/each}}\"}}]}"}

INFO (mergeable): {"logType":"event_received","eventId":"50656700-787d-11eb-90f5-9b0164cf0eed","event":"pull_request.opened","repo":"pr-mergeable","url":"https://github.com/pr-mergeable","isPrivate":false}

Here's how response on app looks like

Seems like nothing is failing, but somehow the response is 202 - still processing

[shine2lay]

@vsingal-p is that all the log you see on the server side? I am expecting to see a log of type logType: "action_after_validate_execute" that should complete the check circuit

Payload delivery is not helpful because the github api we call are separate from response to the event payload

[shine2lay]

To clarify (don't have much experience in node based apps) -

1. re install the node packages -> means running npm install or yarn --ignore-engines .. right?
2. use the correct node and npm version -> means updating engines in package.json and updating .nvmrc file
3. Yes, I had pre [email protected] version

1. yes make sure npm install is ran
2. you should check the installed npm version using npm -v

[quike]

Getting the same error than @vsingal-p

• Server: GitHub Enterprise Server v2.22.7
• Probot: v11.0.1
• Node.js: v14.15.4
• NPM: v6.14.10

ERROR (HttpError): If you would like to help us test the Checks API during its preview period, you must specify a custom media type in the 'Accept' header. Please see the docs for full details.
    HttpError: If you would like to help us test the Checks API during its preview period, you must specify a custom media type in the 'Accept' header. Please see the docs for full details.

status: 415
headers: {
    access-control-allow-origin: "*"
    access-control-expose-headers: "ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset"
    connection: "close"
    content-length: "291"
    content-security-policy: "default-src 'none'"
    content-type: "application/json; charset=utf-8"
    date: "Wed, 10 Mar 2021 17:42:17 GMT"
    referrer-policy: "origin-when-cross-origin, strict-origin-when-cross-origin"
    server: "GitHub.com"
    status: "415 Unsupported Media Type"
    strict-transport-security: "max-age=31536000; includeSubdomains"
    x-content-type-options: "nosniff"
    x-frame-options: "deny"
    x-github-enterprise-version: "2.22.7"
    x-github-media-type: "github.v3; format=json"
    x-github-request-id: "f486a2fe-e589-4ed7-80fe-ef7697aa654b"
    x-runtime-rack: "0.022820"
    x-xss-protection: "1; mode=block"
}

request: {
    method: "POST"
    url: "https://<GITHUB_SERVER>/api/v3/repos/<ORG>/<REPO>/check-runs"
    headers: {
      "accept": "application/vnd.github.v3+json",
      "user-agent": "probot/11.0.1 octokit-core.js/3.3.0 Node.js/14.15.4 (darwin; x64)",
      "authorization": "token [REDACTED]",
      "content-type": "application/json; charset=utf-8"
    }
    body: "{\"status\":\"in_progress\",\"output\":{\"title\":\"Mergeable is running.\",\"summary\":\"Please be patient. We'll get you the results as soon as possible.\"},\"name\":\"Mergeable\",\"started_at\":\"2021-03-10T17:53:58.566Z\",\"head_branch\":\"misc/sso-demo\",\"head_sha\":\"ff0f55666e2b3372e879c0586f7e5a940e76a3a4\"}"
    request: {
      "retryCount": 3,
      "retries": 3,
      "retryAfter": 16
    }
}

So the operation complains of a HTTP Status 415 - Unsupported Media Type.

This is testing from local with a Forwarding https://smee.io webhook

[vsingal-p]

@shine2lay I tried using correct npm version too, didn't work And that are all the logs I could get

Still not working Do github enterprise version has some compatibility issues?

Edit: I am using GitHub Enterprise Server 2.21.6

[quike]

@vsingal-p @shine2lay it is related to probot 11 upgrade.

Current master works fine (although with some deprecation warnings) on the latest v10.19.0 of branch 10.

WARN (Deprecation): [probot] "({ app }) => {}" is deprecated (sorry!). We reverted back to the previous API "(app) => {}", see reasoning at https://github.com/probot/probot/issues/1286#issuecomment-744094299
    Deprecation: [probot] "({ app }) => {}" is deprecated (sorry!). We reverted back to the previous API "(app) => {}", see reasoning at https://github.com/probot/probot/issues/1286#issuecomment-744094299
        at ProbotWithDefaults.get ...
WARN (Deprecation): [probot] "({ app, getRouter }) => {}" is deprecated. Use "(app, { getRouter }) => {}" instead
    Deprecation: [probot] "({ app, getRouter }) => {}" is deprecated. Use "(app, { getRouter }) => {}" instead
        at ProbotWithDefaults.get ...
INFO (probot): Logger Successfully initialized
INFO (mergeable): In PRODUCTION mode.
INFO (mergeable): Scheduler: off!
INFO (server): Running Probot v10.19.0 (Node.js: v14.15.4)
INFO (server): Forwarding https://smee.io/... to http://localhost:3000/
INFO (server): Listening on http://localhost:3000
INFO (server): Connected

The actions checks happen

INFO (action/checks): {"logType":"action_after_validate_execute","eventId":"d4562080-8279-11eb-922c-8bb74dda52e3","repo":"quike/test-mergeable","actionName":"checks","settings":"{\"do\":\"checks\",\"state\":\"completed\",\"status\":\"failure\",\"payload\":{\"title\":\"{{failCount}}/{{validationCount}} Fail(s): {{#each failures}} {{toUpperCase name}}{{^@last}}, {{/@last}}{{/each}}\",\"summary\":\"### Status: {{toUpperCase validationStatus}}\\n\\n        Here are some stats of the run:\\n        {{validationCount}} validations were ran.\\n        {{passCount}} PASSED\\n        {{failCount}} FAILED\\n      \",\"text\":\"{{#each validationSuites}}\\n#### {{{statusIcon status}}} Validator: {{toUpperCase name}}\\n{{#each validations }} * {{{statusIcon status}}} ***{{{ description }}}***\\n       Input : {{{details.input}}}\\n       Settings : {{{displaySettings details.settings}}}\\n       {{/each}}\\n{{/each}}\"}}"}
DEBUG (action/checks): Updating Check for quike/test-mergeable - status=completed
DEBUG (Mergeable):
    owner: "quike"
    repo: "test-mergeable"
    status: "completed"
    output: {...}
    check_run_id: 46330
    conclusion: "failure"
    completed_at: "2021-03-11T14:55:36.908Z"

So something on that 10 to 11 jump doesn't work as expected, at least in GitHub Enterprise (the version I am testing).

UPDATE: It doesn't work on probot 11.0.0 (the first version of the v11)

[vsingal-p]

Nice findings @quike 👍

When you say It doesn't work on probot 11.0.0 (the first version of the v11) .. does this mean it works on newer versions (i.e. 11.1.0) ?

[shine2lay]

@quike good findings, we didn't realize that probot v11 doesn't work for enterprise. if you have found a patch that works, do let us know and i'll try to upgrade it (hopefully it won't break anything else)

[vsingal-p]

Confirmed that it works with 10.19.0 and doesn't work with 11.1.0

[quike]

Nice findings @quike 👍

When you say It doesn't work on probot 11.0.0 (the first version of the v11) .. does this mean it works on newer versions (i.e. 11.1.0) ?

I meant to say that the current mergeable probot v11.0.1 is failing, so I tried the first version of the release 11 (v11.0.0) and it didn't work with that neither. I tested also the last available one v11.1.0 and still the same result.

@shine2lay I haven't got any more time to dig into, but yes, it is not compatible with GitHub Enterprise, because a wrong accept header is being sent.

If you check out the current GitHub Public version for Rest API v3 https://docs.github.com/en/rest/reference/checks#create-a-check-run it says header accept: application/vnd.github.v3+json

but at GitHub Enterprise level https://docs.github.com/en/[email protected]/rest/reference/checks#create-a-check-run accept: application/vnd.github.antiope-preview+json because it is still in preview mode.

On GHE v3 https://docs.github.com/en/[email protected]/rest/reference/checks the api accepts the desired header, but of course, you must wait until your company decide to upgrade.

What intrigues me is why probot 10.19.0 or before works and not probot 11. How is that check run request being generated that even older probot version works on newer servers versions? what was the mechanism before?

[quike]

https://github.com/octokit/rest.js/issues/1983

[jusx]

@quike Out of the box octokit is not fully compatible with GHE. Compatibility with GHE is achieved with additional plugins:

• https://github.com/octokit/plugin-enterprise-server.js
• https://github.com/octokit/plugin-enterprise-compatibility.js

It seems probot is not instantiating octokit with the enterprise-server plugin. See https://github.com/probot/probot/blob/master/src/octokit/probot-octokit.ts

[sambhav]

I am hitting the same issue - is there a workaround?

[vsingal-p]

Downgrading probot to 10.19.0 worked for me

[vsingal-p]

Also verified that the latest master works with Github enterprise 3.0+ 👍