rest.js

rest.js copied to clipboard

I am trying to download an asset from Github. Here is the corresponding code:

import Octokit from "@octokit/rest"
const octokit = new Octokit({
  auth: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
});
const owner = "Geode-solutions";
const repo = "opengeode";
const asset_id = 13450175;
octokit
  .request("GET /repos/:owner/:repo/releases/assets/:asset_id", {
      headers: {
         Accept: "application/octet-stream"
      },
      owner,
      repo,
      asset_id
});

I got this error in return:

OPTIONS https://github-production-release-asset-2e65be.s3.amazonaws.com/156866568/4524ae80-99ab-11e9-93db-8c3a783c062c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T120659Z&X-Amz-Expires=300&X-Amz-Signature=65d90d511f7753dde757d77cab65ccea135868dce37c386e6fdb7a52c1ae813e&X-Amz-SignedHeaders=host&actor_id=3213882&response-content-disposition=attachment%3B%20filename%3DOpenGeode-1.5.16-Linux.tar.gz&response-content-type=application%2Foctet-stream 403 (Forbidden)

Access to fetch at 'https://github-production-release-asset-2e65be.s3.amazonaws.com/156866568/4524ae80-99ab-11e9-93db-8c3a783c062c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T120659Z&X-Amz-Expires=300&X-Amz-Signature=65d90d511f7753dde757d77cab65ccea135868dce37c386e6fdb7a52c1ae813e&X-Amz-SignedHeaders=host&actor_id=3213882&response-content-disposition=attachment%3B%20filename%3DOpenGeode-1.5.16-Linux.tar.gz&response-content-type=application%2Foctet-stream' (redirected from 'https://api.github.com/repos/Geode-solutions/opengeode/releases/assets/13450175') from origin 'null' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

What do I have to add to my headers to make it works? Thanks for your help!

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.54. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

I fear it doesn’t work for a similar reason as https://github.com/octokit/rest.js/issues/758, the CORS headers configured on GitHub’s S3 are probably not allowing for direct downloads in browsers from other domains.

Can you try to not set the custom header so that you can retrieve the browser_download_url from the response. Then try to download it using fetch(browser_download_url, { mode: 'no-cors'})

Let me know if that works

I tried this code but I am not sure about the Authorization section.

fetch(browser_download_url, {
  mode: "no-cors",
  headers: {
     "Authorization": "Bearer xxxxxxxxx"
  }                      
})

Here is the response

GET https://github.com/Geode-solutions/OpenGeode/releases/download/v1.5.16/OpenGeode-1.5.16-Linux.tar.gz 404 (Not Found)

Response {type: "opaque", url: "", redirected: false, status: 0, ok: false, …}

As this is in the browser, would it be a redirection to just redirect to https://github.com/Geode-solutions/OpenGeode/releases/download/v1.5.16/OpenGeode-1.5.16-Linux.tar.gz or whatever browser_download_url returns?

I’ll look more into it, I’m curious where exactly the problem lies, but it’s most likely a problem with CORS settings on Amazon.

Could you explain your use case why you need to download the asset in the browser? I’m trying to build a case to have these CORS headers added, knowing your use case will help :)

browser_download_url returns https://github.com/Geode-solutions/OpenGeode/releases/download/v1.5.16/OpenGeode-1.5.16-Linux.tar.gz, so ok on this side.

I am building an electron app with a plugin system. Theses plugins use custom native node modules. These modules are already compiled and released as in a Github release asset (one for each platform: windows, linux, mac). My goal is to automatically download these "ready to go" assets from the plugin.

I’m not very familiar with Electron, but I don’t think it should be affected by CORS limitations as its a native app? Could you maybe create a repository with a minimal electron app to reproduce the problem?

And could you test it against a public repository? Does it work there?

It works with a public repository. I must have my headers wrong. I have a personal token, how should I use it in the fetch?

In fetch you’d pass the personal access token like this

fetch(url, {
  headers: {
    authorization: 'token <YOUR TOKEN HERE>'
  }
})

Ok, so I tried this code

fetch(browser_download_url, {
  headers: {
    authorization: "token xxxxxxxx"
  }
})

Here the response

Response {
  size: 0,
  timeout: 0,
  [Symbol(Body internals)]:
   { body:
      Gunzip {
        _writeState: [Uint32Array],
        _readableState: [ReadableState],
        readable: true,
        _events: [Object],
        _eventsCount: 7,
        _maxListeners: undefined,
        _writableState: [WritableState],
        writable: true,
        allowHalfOpen: true,
        _transformState: [Object],
        _hadError: false,
        bytesWritten: 0,
        _handle: [Zlib],
        _outBuffer:
         <Buffer 4e 6f 74 20 46 6f 75 6e 64 83 cb e7 55 7f 00 00 10 5b 87 02 00 00 00 00 10 5b 87 02 00 00 00 00 d0 58 87 02 00 00 00 00 f8 ed 88 02 00 00 00 00 0b 00 ... >,
        _outOffset: 0,
        _chunkSize: 16384,
        _defaultFlushFlag: 2,
        _finishFlushFlag: 2,
        _nextFlush: -1,
        _defaultFullFlushFlag: 3,
        _info: undefined,
        _level: -1,
        _strategy: 0 },
     disturbed: false,
     error: null },
  [Symbol(Response internals)]:
   { url:
      'https://github.com/Geode-solutions/OpenGeode/releases/download/v2.2.0/OpenGeode-2.2.0-Linux.tar.gz',
     status: 404,
     statusText: 'Not Found',
     headers: Headers { [Symbol(map)]: [Object] },
     counter: 0 } }

The same token is used with octokit and it works to call the github API.

@gr2m Any idea of why it does not work? Are you able to reproduce this issue?

I was able to reproduce the problem. I’ve created a test repository and invited you to it: https://github.com/gr2m/octokit-rest.js-1417/invitations

I have no workaround I’m afraid. This is a limitation with the CORS settings on GitHub’s S3 as far as I can tell. I’ll investigate some more and then bring it up with GitHub’s support/security team. There are a few other use cases where missing CORS settings is a problem for usage in browser: https://github.com/octokit/rest.js/issues?q=is%3Aissue+is%3Aopen+label%3A%22blocked+by+api%22

Looks like this does not throw an error

const assetUrl = https://api.github.com/repos/gr2m/octokit-rest.js-1417/releases/assets/13689345
const token = 'your token here'
const response = await fetch(`${assetUrl}?access_token=${token}`, {
  headers: {
    accept: 'application/octet-stream'
  },
  mode: 'no-cors'
})

but response.body is null. My understanding is that no-cors means you cannot get access to the returned data at all

@gr2m any update or news from GitHub’s support/security team?

I’ll bring it up tomorrow in a meeting, I’ll keep you posted here.

Any feedback from your meeting?

Don't expect this to be resolved anytime soon. I will update this issues as soon as there are any news.

Any news?

Unfortunately no. The only thing you can do right now is to inform [email protected] that this is a blocking issue for you to bump the priority on the problem internally.

@gr2m any update? Github is deprecating the workarround with access_token: https://developer.github.com/changes/2019-11-05-deprecated-passwords-and-authorizations-api/#authenticating-using-query-parameters

No :( I've let them know but didn't hear back. I'll have a call with some folks of the API team tomorrow and will bring it up directly, I'll keep you posted

Hello @gr2m, any update on this?

I promise that as soon as there is an update I will post it here. For the time being, the best you can do is to contact support at https://support.github.com/contact and let them know about your use case. They are aware of the problem, but the more people report it and share their use cases, the higher the priority will get.

Thank you all for your patience and support

Hi, have you tried with:

request(
  {
    url:
      "https://api.github.com/repos/:owner/:repo/releases/assets/:asset_id",
    method: "GET",
    headers: {
      Accept: "application/octet-stream",
      Authorization: "token " + process.env.GITHUB_TOKEN,
      "User-Agent": "",
    },
  },
  function (error, response, body) {
    console.log(body);
  }
);

The body should contains the content of your asset.

You can do it also with curl:

curl -v -L -H 'Accept: application/octet-stream' -H 'Authorization: token YOUR_TOKEN' https://api.github.com/repos/:owner/:repo/releases/assets/:asset_id

This won't for private assets that redirect to s3. AWS complains that the Authorization header is present. You can use the query param to work around this, but it looks like that support is being deprecated.

This won't for private assets that redirect to s3. AWS complains that the Authorization header is present. You can use the query param to work around this, but it looks like that support is being deprecated.

I'm using the above code to retrive private assets that redirect to S3 and it works for me.

The -L params in curl do the job.

In the nodeJS part, request follows automatically the redirects (for GET requests) but you have to add:

"User-Agent": ""

because without you get this error:

Request forbidden by administrative rules. Please make sure your request has a User-Agent header (http://developer.github.com/v3/#user-agent-required). Check https://developer.github.com for other possible causes.

Have you try it? Let me know.

Not working for me either with private repos. I really need that to provide assets to a landing page from a private repo, do you have any update ?

No update. But I created a dedicated repository for GitHub API limitations, including the use cases that GitHub users have for them. I hope that will help with the decision making on GitHub's site.

Please add your use cases to https://github.com/gr2m/github-api-wishlist/tree/master/wishlist/cors-for-adjacent-domains

Thanks @edospadoni for the hint!

For now, we have to download assets from private repos via plain HTTP calls outside Octokit. I use got for the HTTP call -

// TypeScript implementation:
import got from "got";
import { pipeline } from "stream";
import { createWriteStream } from "fs";
import { promisify } from "util";
const asyncPipeline = promisify(pipeline);

type TDownload = {
  owner: string;
  repo: string;
  assetId: number;
  toLocalFile: string;
  githubToken?: string;
};

export async function downloadGithubAsset(dl: TDownload) {
  const token = dl.githubToken || process.env.GITHUB_TOKEN;
  return asyncPipeline(
    got.stream(
      `https://api.github.com/repos/${dl.owner}/${dl.repo}/releases/assets/${dl.assetId}`,
      {
        method: "GET",
        headers: {
          Accept: "application/octet-stream",
          Authorization: `token ${token}`,
          "User-Agent": "",
        },
      }
    ),
    createWriteStream(dl.toLocalFile)
  );
}

Based on https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/, access_token will be removed 5th of May so it would be great if Octokit could support auth for private release assets by then.

My understanding is that a part of the issue is that fetch is handling redirect transparently within its implementation and then passing the auth headers to the target of the redirect resulting in an error.

I imagine in ideal case the fix would be on server-side so that fetch would work and possibly that would fix Octokit as well as a side effect.

My use case is downloading private assets at a Next.js site and then serving them through it. I'm publishing private fonts this way by using GitHub releases as a registry and Next.js as my proxy.

For what it's worth, here's an Axios based solution that seems to work:

async function getAsset(url) {
  const response = await axios.get(url, {
    headers: {
      Accept: "application/octet-stream",
      Authorization: `token ${process.env.GITHUB_TOKEN}`,
      "User-Agent": ""
    },
    responseType: "arraybuffer"
  });

  return response.data;
}

It expects an url from an asset object gained through GitHub API as a parameter.