Unable to use chucked media upload / mediaUploadAppend: "Could not authenticate you"

[slorber]

Describe the bug

The chunked media upload is required to upload videos, but it seems impossible to use currently due to an auth failure.

Unable to authenticate on the 2nd / APPEND endpoint of the chunked media upload.

  const mediaUploadInitResult = await twitterClient.media.mediaUploadInit({
    command: "INIT",
    media_type: "video/mp4",
    media_category: "tweet_video",
    total_bytes: 56789710,
  });
  console.log("mediaUploadInitResult", mediaUploadInitResult);

  const binary = fs.readFileSync(filePath);

  const base64 = fs.readFileSync(filePath, { encoding: "base64" });

  const mediaUploadAppend = await twitterClient.media.mediaUploadAppend({
    command: "APPEND",
    media_id: mediaUploadInitResult.media_id_string,
    media_data: base64,
    // media: binary,
    segment_index: 0,
  });
  console.log("mediaUploadAppend", mediaUploadAppend);

I tried both with binary or base64 and it does not change anything. Note non-chunked image upload works fine for me.

mediaUploadInitResult {
  media_id: 1374383257052012500,
  media_id_string: '1374383257052012546',
  expires_after_secs: 86399,
  media_key: '7_1374383257052012546'
}
Error
{
  statusCode: 401,
  data: '{"errors":[{"code":32,"message":"Could not authenticate you."}]}'
}
error Command failed.

The first INIT call works, but the 2nd APPEND call fails.

It looks like a problem related to how the OAuth signature is handled for multipart uploads, according to this blog post: https://retifrav.github.io/blog/2019/08/22/twitter-chunked-upload-video/

This page also mentions:

I believe there may be something wrong that prevents chunked upload in the transport layer here: https://github.com/FeedHive/twitter-api-client/blob/master/src/base/Transport.ts

There are not many examples on the internet using NodeJS and the official doc is using twurl unfortunately. This could be helpful: https://medium.com/ameykpatil/how-to-publish-an-external-video-to-twitter-node-js-version-89c03b5ff4fe

Would be happy to help solve this

[vsnthdev]

I believe media_data and command fields are taken care by the library, so you wouldn't have to provide those...

[slorber]

Media data is the data of the file, so if I don't provide it how would it work?

Not passing command lead to an error even for Init.

[SimonHoiberg]

Hi @slorber Twitter has some unfortunate error messages, because this one "Could not authenticate you." means that something is wrong with the encoding of your file in 90% of the cases. Most likely, it doesn't have anything to do with authentication.

We are using the chunked media uploads in FeedHive for both images, gifs, and video, so I know for a fact that it works. But I can't tell why it's not working for you in this case, unfortunately.

However, we are actually looking into making the whole process of uploading media through chunked an inbuilt part of the library, so people won't have to do all the gymnastics themselves in the future.

[slorber]

Hi @SimonHoiberg

I'm not sure to understand what you mean here:

• is it a problem in the source video encoding?
• is it a problem not reading binary/base64 correctly?

I tried chunked upload using a .mp4 file I downloaded on your Feedhive twitter account, and also tried with a regular image, and it does not work for me.

I also believe that fs.readFileSync(filepath) reads binary correct and fs.readFileSync(filePath, { encoding: "base64" }); reads base64 correctly.

Would you mind sharing a code snipped you use to make this work in Feedhive? It drives me crazy as I have no idea what I'm doing wrong here.

[slorber]

For what it's worth, I'm able to upload the video using another lib. But I would prefer using one lib instead of 2 😅

const Twitter = require("twitter");

const client = new Twitter({
  // ... auth
});

const filePath = "/Users/sebastienlorber/Desktop/video.mp4";

const mediaType = "video/mp4";
const mediaData = require("fs").readFileSync(filePath);
const mediaSize = require("fs").statSync(filePath).size;

initUpload() // Declare that you wish to upload some media
  .then(appendUpload) // Send the data for the media
  .then(finalizeUpload) // Declare that you are done uploading chunks
  .then((mediaId) => {
    console.log("mediaId", mediaId);
    // You now have an uploaded movie/animated gif
    // that you can reference in Tweets, e.g. `update/statuses`
    // will take a `mediaIds` param.
  });

/**
 * Step 1 of 3: Initialize a media upload
 * @return Promise resolving to String mediaId
 */
function initUpload() {
  return makePost("media/upload", {
    command: "INIT",
    total_bytes: mediaSize,
    media_type: mediaType,
  }).then((data) => {
    console.log("INIT data", data);
    return data.media_id_string;
  });
}

/**
 * Step 2 of 3: Append file chunk
 * @param String mediaId    Reference to media object being uploaded
 * @return Promise resolving to String mediaId (for chaining)
 */
function appendUpload(mediaId) {
  return makePost("media/upload", {
    command: "APPEND",
    media_id: mediaId,
    media: mediaData,
    segment_index: 0,
  }).then((data) => {
    console.log("APPEND data", data);
    return mediaId;
  });
}

/**
 * Step 3 of 3: Finalize upload
 * @param String mediaId   Reference to media
 * @return Promise resolving to mediaId (for chaining)
 */
function finalizeUpload(mediaId) {
  return makePost("media/upload", {
    command: "FINALIZE",
    media_id: mediaId,
  }).then((data) => {
    console.log("FINALIZE data", data);
    return mediaId;
  });
}

/**
 * (Utility function) Send a POST request to the Twitter API
 * @param String endpoint  e.g. 'statuses/upload'
 * @param Object params    Params object to send
 * @return Promise         Rejects if response is error
 */
function makePost(endpoint, params) {
  return new Promise((resolve, reject) => {
    client.post(endpoint, params, (error, data, response) => {
      if (error) {
        reject(error);
      } else {
        resolve(data);
      }
    });
  });
}

[slorber]

Any the exact same code with your lib fails, no matter how I try to provide the file data:

const { twitterClient } = require("./lib/twitterClient");

const filePath = "/Users/sebastienlorber/Desktop/video.mp4";

const mediaType = "video/mp4";
const mediaData = require("fs").readFileSync(filePath);
const mediaSize = require("fs").statSync(filePath).size;

initUpload() // Declare that you wish to upload some media
  .then(appendUpload) // Send the data for the media
  .then(finalizeUpload) // Declare that you are done uploading chunks
  .then(
    (mediaId) => {
      console.log("mediaId", mediaId);
      // You now have an uploaded movie/animated gif
      // that you can reference in Tweets, e.g. `update/statuses`
      // will take a `mediaIds` param.
    },
    (e) => console.log(e)
  );

/**
 * Step 1 of 3: Initialize a media upload
 * @return Promise resolving to String mediaId
 */
function initUpload() {
  return twitterClient.media
    .mediaUploadInit({
      command: "INIT",
      total_bytes: mediaSize,
      media_type: mediaType,
    })
    .then((data) => {
      console.log("INIT data", data);
      return data.media_id_string;
    });
}

/**
 * Step 2 of 3: Append file chunk
 * @param String mediaId    Reference to media object being uploaded
 * @return Promise resolving to String mediaId (for chaining)
 */
function appendUpload(mediaId) {
  return twitterClient.media
    .mediaUploadAppend({
      command: "APPEND",
      media_id: mediaId,
      // media: mediaData,
      // media: mediaData.toString(),
      // media_data: mediaData.toString("base64"),
      media_data: require("fs").readFileSync(filePath, { encoding: "base64" }),
      segment_index: 0,
    })
    .then((data) => {
      console.log("APPEND data", data);
      return mediaId;
    });
}

/**
 * Step 3 of 3: Finalize upload
 * @param String mediaId   Reference to media
 * @return Promise resolving to mediaId (for chaining)
 */
function finalizeUpload(mediaId) {
  return twitterClient.media
    .mediaUploadFinalize({
      command: "FINALIZE",
      media_id: mediaId,
    })
    .then((data) => {
      console.log("FINALIZE data", data);
      return mediaId;
    });
}

Hope this will be helpful to debug the issue, in the meantime I'll just use 2 libs 😅

[SimonHoiberg]

Thanks a lot, we'll take a look :blush:

[TotomiEcio]

Hi! I'm still having the same issues with the APPEND request. Have you made any progress?

[jucasoliveira]

Hi! I'm still having the same issues with the APPEND request. Have you made any progress?

I had an issue too, then I saw the sement_index to be passing a number, It must be a string

/**
   * Step 2 of 3: Append file chunk
   * @param String mediaId    Reference to media object being uploaded
   * @return Promise resolving to String mediaId (for chaining)
   */
  async function appendUpload(mediaId: any) {
    const data = await twitterClient.media.mediaUploadAppend({
      command: 'APPEND',
      media_id: mediaId,
      // media: mediaData,
      // media: mediaData.toString(),
      // media_data: mediaData.toString("base64"),
      media_data: require('fs').readFileSync(mediaPath, { encoding: 'base64' }),
      segment_index: '0'
    });
    console.log('APPEND data', data);
    return mediaId;
  }

@slorber code helped me, thanks!

[aditodkar]

Hi @slorber @jucasoliveira @SimonHoiberg can you please help or suggest why this node.js script does not work?

code:

const axios = require('axios');
const { TwitterClient } = require('twitter-api-client');
const mediaType = "video/mp4";

const twitterClient = new TwitterClient({
    apiKey: '',
    apiSecret: '',
    accessToken: '',
    accessTokenSecret: '',
});

const downloadImageFromUrl = async () => {
    const video = await axios.get(`https://i.imgur.com/rfYwI5n.mp4`, { responseType: 'arraybuffer' });
    const buffer = Buffer.from(video.data, 'binary').toString('base64');
    return buffer;
}

let mediaSize = 0;
let mediaFile;

async function main() {
    mediaFile = await downloadImageFromUrl();
    mediaSize = mediaFile.length;

    initUpload() // Declare that you wish to upload some media
        .then(appendUpload) // Send the data for the media
        .then(finalizeUpload) // Declare that you are done uploading chunks
        .then(
            (mediaId) => {
                console.log("mediaId", mediaId);
                // You now have an uploaded movie/animated gif
                // that you can reference in Tweets, e.g. `update/statuses`
                // will take a `mediaIds` param.
            },
            (e) => console.log(e)
        );
}

main();


/**
 * Step 1 of 3: Initialize a media upload
 * @return Promise resolving to String mediaId
 */
function initUpload() {
    return twitterClient.media
        .mediaUploadInit({
            command: "INIT",
            total_bytes: mediaSize,
            media_type: mediaType,
        })
        .then((data) => {
            console.log("INIT data", data);
            return data.media_id_string;
        });
}

/**
 * Step 2 of 3: Append file chunk
 * @param String mediaId    Reference to media object being uploaded
 * @return Promise resolving to String mediaId (for chaining)
 */
function appendUpload(mediaId) {
    return twitterClient.media
        .mediaUploadAppend({
            command: "APPEND",
            media_id: mediaId,
            media_data: mediaFile,
            segment_index: "0",
        })
        .then((data) => {
            console.log("APPEND data", data);
            return mediaId;
        });
}

/**
 * Step 3 of 3: Finalize upload
 * @param String mediaId   Reference to media
 * @return Promise resolving to mediaId (for chaining)
 */
function finalizeUpload(mediaId) {
    return twitterClient.media
        .mediaUploadFinalize({
            command: "FINALIZE",
            media_id: mediaId,
        })
        .then((data) => {
            console.log("FINALIZE data", data);
            return mediaId;
        });
}

Above code is not working any idea why so? Is it because I am not generating file buffer correctly?

[aditodkar]

Hi @slorber @jucasoliveira @SimonHoiberg can you please help or suggest why this node.js script does not work?

code:

const axios = require('axios');
const { TwitterClient } = require('twitter-api-client');
const mediaType = "video/mp4";

const twitterClient = new TwitterClient({
    apiKey: '',
    apiSecret: '',
    accessToken: '',
    accessTokenSecret: '',
});

const downloadImageFromUrl = async () => {
    const video = await axios.get(`https://i.imgur.com/rfYwI5n.mp4`, { responseType: 'arraybuffer' });
    const buffer = Buffer.from(video.data, 'binary').toString('base64');
    return buffer;
}

let mediaSize = 0;
let mediaFile;

async function main() {
    mediaFile = await downloadImageFromUrl();
    mediaSize = mediaFile.length;

    initUpload() // Declare that you wish to upload some media
        .then(appendUpload) // Send the data for the media
        .then(finalizeUpload) // Declare that you are done uploading chunks
        .then(
            (mediaId) => {
                console.log("mediaId", mediaId);
                // You now have an uploaded movie/animated gif
                // that you can reference in Tweets, e.g. `update/statuses`
                // will take a `mediaIds` param.
            },
            (e) => console.log(e)
        );
}

main();


/**
 * Step 1 of 3: Initialize a media upload
 * @return Promise resolving to String mediaId
 */
function initUpload() {
    return twitterClient.media
        .mediaUploadInit({
            command: "INIT",
            total_bytes: mediaSize,
            media_type: mediaType,
        })
        .then((data) => {
            console.log("INIT data", data);
            return data.media_id_string;
        });
}

/**
 * Step 2 of 3: Append file chunk
 * @param String mediaId    Reference to media object being uploaded
 * @return Promise resolving to String mediaId (for chaining)
 */
function appendUpload(mediaId) {
    return twitterClient.media
        .mediaUploadAppend({
            command: "APPEND",
            media_id: mediaId,
            media_data: mediaFile,
            segment_index: "0",
        })
        .then((data) => {
            console.log("APPEND data", data);
            return mediaId;
        });
}

/**
 * Step 3 of 3: Finalize upload
 * @param String mediaId   Reference to media
 * @return Promise resolving to mediaId (for chaining)
 */
function finalizeUpload(mediaId) {
    return twitterClient.media
        .mediaUploadFinalize({
            command: "FINALIZE",
            media_id: mediaId,
        })
        .then((data) => {
            console.log("FINALIZE data", data);
            return mediaId;
        });
}

Above code is not working any idea why so? Is it because I am not generating file buffer correctly?

Instead directly using image url and encoding it. I tried one more thing. First I downloaded the image/video using one node script:

const https = require('https');
const fs = require('fs');

function saveImageToDisk(url, path) {
    const localpath = fs.createWriteStream(path);

    const response = https.get(url, function (res) {
        res.pipe(localpath);
    });

    console.log("response", response);
}

saveImageToDisk('https://i.imgur.com/rfYwI5n.mp4', "hello" + ".mp4")

console.log("done")

And once the image is downloaded I mentioned same file path to above twitterClient node script and I was successfully able to generate media id for it. One thing I am not able to understand is this issue specific to twitter or twiiter-api-client package or am I doing something wrong with nodejs file buffer? Please suggest. Thanks