drive.files.create creates a file with 0 bytes

[tapz]

Environment details

• OS: macOS 11.4
• Node.js version: 14.16.1
• npm version: 6.14.12
• googleapis version: 78.0.0

Description

Files uploaded to Google Drive using a stream or buffer have length of zero bytes. No errors anywhere. Works only when body is a string, but how do you make a string from a binary file and how do you specify the charset for a text file?

Steps to reproduce

const { google } = require('googleapis');
const Readable = require('stream').Readable;

async function uploadToGoogle(googlePrivateKey) {
  const scopes = [
    'https://www.googleapis.com/auth/drive'
  ];

  const auth = new google.auth.JWT(
    ***********@**********.iam.gserviceaccount.com', null,
    googlePrivateKey, scopes
  );

  const drive = google.drive({ version: 'v3', auth });

  const buffer = Buffer.from('Test data 1', 'latin1');
  
  const driveFile = await drive.files.create({
    requestBody: {
      name: 'testfile.txt',
      mimeType: 'text/plain',
      parents: [
        '***************************************'
      ],
      supportsAllDrives: true
    },
    media: {
      mimeType: 'text/plain',
      body: new Readable(buffer)
    }
  });
}

[bcoe]

@tapz do both strings and Buffers work when the file is non-zero in length?

[tapz]

@bcoe Only strings. Giving a Buffer instead of a stream causes a zero length file too.

[tapz]

As a workaround, I implemented my own multipart upload with binary transfer encoding using Axios. Just use the google lib to do the authentication and put res.config.headers.Authorization to Authorization header. Axios is able to handle the given Buffers.

[Rafahur03]

Hi @tapz , I'm new to programming, can you explain your solution? I need upload images to google drive but I have the same that you, Excuse me for my English

[tapz]

@Rafahur03

const axios = require('axios');
const { v4: uuidv4 } = require('uuid');

const { google } = require('googleapis');

const drive = google.drive({ version: 'v3', auth });

const res = await drive.files.list({
  q: `'${rootFolderId}' in parents`,
  pageSize: '1000',
  supportsAllDrives: 'true',
});

const folderId = res.data.files.find(f => f.name === folderName)?.id;

const boundary = uuidv4();
const jsonBody = JSON.stringify({
  name: filename,
  parents: [folderId]
});

const bodyHead = 
  `--${boundary}\r\n` + 
  'content-type: application/json\r\n' + 
  '\r\n' +
  `${jsonBody}\r\n` + 
  `--${boundary}\r\n` + 
  `content-type: ${mimeType}\r\n` + 
  'content-transfer-encoding: binary\r\n' +
  `content-length: ${buffer.length}\r\n` +
  '\r\n';

const bodyTail = `\r\n--${boundary}--`;

const response = await axios.post(
  'https://www.googleapis.com/upload/drive/v3/files?fields=id&uploadType=multipart', 
  Buffer.concat([
    Buffer.from(bodyHead),
    buffer,
    Buffer.from(bodyTail)
  ]), {
  headers: {
    'content-type': `multipart/related; boundary=${boundary}`,
    Authorization: res.config.headers.Authorization,
    Accept: 'application/json'
  }
});

[Rafahur03]

@tapz I appreciate very much your help, I see that my knowledge is still very limited, I understand part of the code, you are consulted all the folders of your drive select one and then create the headers and the body of the requirement that will be made to the url of drive, I do not see where the file buffer is being loaded and in my case I do not need to find a folder and I see that the response of that query gets authorization data in my case, can I replace it with const auth = new google.auth.GoogleAuth({ keyFile, scopes })

headers: { 'content-type': multipart/related; boundary=${boundary}, Authorization: auth, Accept: 'application/json' }

[OlivierChirouze]

@tapz, you saved my day!

[karchx]

A good solution he found was to use the temp Path that gives form-data when uploading a file and use the createReadStream of the fileSystem

const uploadFilesAuth = async (fileObject, folderId) => {
  const drive = google.drive({ version: 'v3', auth });

  const fileMetadata = {
    name: fileObject.name,
    parents: [folderId],
  };
  const media = {
    mimeType: fileObject.mimeType,
    body: fs.createReadStream(fileObject.tempFilePath)
  };

  try {
    const file = await drive.files.create({
      resource: fileMetadata,
      media,
      fields: 'id',
    });
    return file.data.id;
  } catch (err) {
    throw err;
  }
}

[sidsethupathi]

Going to give @tapz's axios try a suggestion, but wanted to provide one data point I'm seeing - unsure if it's related to the same issue. The behavior seems similar though: I'm seeing some files upload with 0 bytes when using the v3.drive.files.create function. I also am having a hard time reproducing this locally, it appears to only happen when running in GCE (and even then it looks like only 1/2 the files fail to create). v3.drive.files.update seems to work consistently.

Instead of using Buffer I'm using fs.createReadStream.

I logged out the failed request response (the one that uploaded 0 bytes) and noticed a couple differences when compared to the response of a successful request.

The failed request was missing the following headers:

'x-goog-api-client': 'gdcl/7.0.1 gl-node/18.20.0',
'Accept-Encoding': 'gzip',
'User-Agent': 'google-api-nodejs-client/7.0.1 (gzip)',

The failed request also had a differently shaped body data logged out.

data: '--a21d7000-3191-477c-af5c-df126a8f1048\\r\\n' +
  'content-type: application/json\\r\\n' +
  '\\r\\n' +
  `{\"name\":\"File name\",\"description\":\"File description\",\"parents\":[\"redacted_drive_id\"]}\\r\\n` +
  '--a21d7000-3191-477c-af5c-df126a8f1048\\r\\n' +
  'content-type: video/mp4\\r\\n' +
  '\\r\\n' +
  '--a21d7000-3191-477c-af5c-df126a8f1048--',

This looks like the actual content is missing from this stringified multipart request. Hopefully using axios directly will expose whether this is an issue with the client or my usage of fs.createReadStream.

The successful request has this in the data field:

data: PassThrough {
   _readableState: [ReadableState],
  _events: [Object: null prototype],
  _eventsCount: 2,
  _maxListeners: undefined,
  _writableState: [WritableState],
  allowHalfOpen: true,
  _flush: [Function: flush],
  [Symbol(kCapture)]: false,
  [Symbol(kCallback)]: null
},