s3.getObject Promise example

[kaihendry]

http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/requests-using-stream-objects.html is not a super useful code snippet in light of the way folks use Promises nowadays.

It took me far too long to discover a working example like:

function downloadImage (key) {
  return new Promise((resolve, reject) => {
    const destPath = `/tmp/${path.basename(key)}`
    const params = { Bucket: 'EXAMPLE', Key: key }
    s3.getObject(params)
      .createReadStream()
      .pipe(fs.createWriteStream(destPath))
      .on('close', () => {
        resolve(destPath)
      })
  })
}

Be good if this could be added to the documentation? Many thanks,

[chrisradek]

@kaihendry Thanks for the suggestion. Tagging this issue with documentation for tracking.

For the example you wrote, you might want to listen for the error events on either stream so you can decide how to handle them as well.

function downloadImage (key) {
  return new Promise((resolve, reject) => {
    const destPath = `/tmp/${path.basename(key)}`
    const params = { Bucket: 'EXAMPLE', Key: key }
    const s3Stream = s3.getObject(params).createReadStream();
    const fileStream = fs.createWriteStream(destPath);
    s3Stream.on('error', reject);
    fileStream.on('error', reject);
    fileStream.on('close', () => { resolve(destPath);});
    s3Stream.pipe(fileStream);
  });
}

[cvrajeesh]

@chrisradek shouldn't we close the stream on error in other stream?

[antonsamper]

@kaihendry You're mixing promises and streams. Why dont you just return a promise using the SDK using the promise() method?

s3.getObject(params).promise().then(...).catch(...)

[KidA001]

For anyone using async await:

async function getS3File(filename) {
  const params = {
    Bucket: 'some-bucket-name',
    Key: filename,
    Body: fileIO,
  };

  const response = await s3.getObject(params, (err) => {
    if (err) {
      // handle errors
    }
  });

  return response.Body.toString(); // your file as a string
}

Note you can return response.Body to work with the buffer directly.

If you want the function to return a promise just do this instead:

function getS3File(filename) {
  const params = {
    Bucket: 'some-bucket-name',
    Key: filename,
    Body: fileIO,
  };

  return s3.getObject(params, (err) => {
    if (err) {
      // handle errors
    }
  }).promise();
}

[steve-rhodes]

@antonsamper But what if I want to have a promise for a stream? I'm really struggling to figure this out!

Any idea?

Right now I have:

s3.getObject(params).createReadStream().pipe(file);

How would I be able to catch any errors or know that the stream/write has completed?

Thanks!

[chrisradek]

@elasticsteve You would need to wrap the stream in a promise, like in this example.

Note: like @cvrajeesh commented, you would want to take care to properly close streams in the error events before calling reject, but the example is a good starting point.

[chrisradek]

@elasticsteve You can totally just pass the resolve function to the close event. My example was for the original poster, and they were returning the destPath so my example did as well.

Edit: Realized I didn’t explain what it is doing though, just why. If you were to await the returned promise (or chain then on it), you would get the destination path returned to you.

[steve-rhodes]

@chrisradek Thanks and sorry, I deleted my question just before you posted the reply, because all of a sudden I understood why things were that way.

[steve-rhodes]

@chrisradek Sorry for being a pain, but how do you close the S3 stream?

Would this work (I don't think s3 has a destroy() method):

function downloadFile(key,destPath) {
    return new Promise((resolve, reject)=>{
        const params = {
            Bucket    : '/xxxx',
            Key    : key
        };
        const s3Stream = s3.getObject(params).createReadStream();
        const fileStream = fs.createWriteStream(destPath);
        s3Stream.on('error', ()=>{
            fileStream.destroy();
            reject();
        });
        fileStream.on('error', ()=>{
            s3Stream.destroy();
            reject();
        });
        fileStream.on('close', resolve);
        s3Stream.pipe(fileStream);
    });
}

[deejbee]

@KidA001 I'm confused with the example you gave:

const response = await s3.getObject(params, (err) => {
    if (err) {
      // handle errors
    }
  });

  return response.Body.toString(); // your file as a string

since the aws docs seem to suggest it returns an AWS.Request object which must be set up to handle the relevant events.

Does the example you mentioned refer to a very recent version of the api or is there something i'm not understanding?

[DrMiaow]

The issue is the mixing of Promise and stream semantics. A Promise must resolve() or reject(reason). A stream can fail at any point for any number of reasons, even if a stream is intially returned.

If you have a Promise that returns a stream, you still need to monitor it for errors outside the context of the promise.

So, my approach to resolving this is ...

Inside the promise...

try {
  resolve(s3.getObject(params).createReadStream())
} catch (err) {
  reject(new Error(err.message))
}

When invoking the promise

....then((stream) => {
  stream.on('error', ()=>{
    // Handle error            
  })
  // Do stuff   
}

[a1anw2]

Is there anything special you need to do to your Lambda function to make this example at the top of the thread work? There is nothing I can do to make this work.

It works beautifully when i run it locally via the serverless framework.

No matter what I do, this will simply not download a single byte when run as a Lambda function.

function downloadImage (s3Input) {
  return new Promise((resolve, reject) => {
    const destPath = "/tmp/tabbedFile.csv";
    
    const s3Stream = S3.getObject(s3Input).createReadStream();
    const fileStream = fs.createWriteStream(destPath);
    s3Stream.on("error", reject);
    fileStream.on("error", reject);
    fileStream.on("close", () => { resolve(destPath);});
    s3Stream.pipe(fileStream);
  });
}

I have attempted async/wait .. as well as .then(). So i guess if someone can give me a complete example of a Lambda of this running, that would help a lot! thank you

[skipjack]

I stumbled on this thread as I was surprised no official promise support exists yet. However, in my case I didn't need a stream or to download the file locally. So here's slightly simpler option for those who just want a s3 getObject they can await:

/**
 * Get an object from a s3 bucket
 * 
 * @param  {string} key - Object location in the bucket
 * @return {object}     - A promise containing the response
 */
const getObject = key => {
    return new Promise((resolve, reject) => {
        s3.getObject({
            Bucket: process.env.BUCKET_NAME, // Assuming this is an environment variable...
            Key: key
        }, (err, data) => {
            if ( err ) reject(err)
            else resolve(data)
        })
    })
}

And a small usage example:

async () => {
    try {
        // You'd probably replace 'someImage' with a variable/parameter
        const response = await getObject('someImage.jpg')

    } catch (err) {
        console.error(err)
    }
}

Hoping this package adds first-class support for promises soon though!

[greg-benner-klick-sensei]

I think it does offer promise support @skipjack as per https://github.com/aws/aws-sdk-js/issues/1436#issuecomment-375101246 You just add .promise to the s3. call So you can just do

await s3.getObject({
            Bucket: process.env.BUCKET_NAME, // Assuming this is an environment variable...
            Key: key
    }).promise()

[greg-benner-klick-sensei]

actually seems to be better your way, the promise method they provide seems flaky

[DaveCollinsJr]

Anyone still looking at this... With Nodejs 8.10, this is a simple working version I've used to get an S3 object to the Lambda's /tmp directory:

  const getObject = (handle) => {
    return new Promise((resolve, reject) => {
      s3.getObject(handle, (err, data) => {
        if (err) reject(err)
        else resolve(data.Body)
      })
    })
  };

Then:

var handle = {Bucket: root_bucket, Key: source_file};
const file_data = await getObject(handle);
// file_data is the actual file data... You can fs.writeFile that to your /tmp directory

[rajeshdavidbabu]

@kaihendry You're mixing promises and streams. Why dont you just return a promise using the SDK using the promise() method?

s3.getObject(params).promise().then(...).catch(...)

How will you createAStream and close one in this approach ??

[johncmunson]

@KidA001 you forgot to chain .promise() onto s3.getObject. See below for the correction.

For anyone using async await:

async function getS3File(filename) {
  const params = {
    Bucket: 'some-bucket-name',
    Key: filename,
    Body: fileIO,
  };

  const response = await s3.getObject(params, (err) => {
    if (err) {
      // handle errors
    }
  }).promise();

  return response.Body.toString(); // your file as a string
}

[BrandonCopley]

@johncmunson I've added this code to try aws-sdk with promises, but I get the following error:

"TypeError: Cannot read property 'push' of undefined"

at Request.HTTP_DATA (node_modules/aws-sdk/lib/event_listeners.js:381:35) at Request.callListeners (node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (node_modules/aws-sdk/lib/request.js:683:14) at IncomingMessage.onReadable (node_modules/aws-sdk/lib/event_listeners.js:281:32) at IncomingMessage.emit (events.js:189:13) at IncomingMessage.EventEmitter.emit (domain.js:441:20) at emitReadable_ (_stream_readable.js:535:12) at process._tickCallback (internal/process/next_tick.js:63:19)

[BrandonCopley]

This appears to be happening because the AWS-SDK getObject code was wrapping in an async iterator. Are async/await methods being worked on on this project?

[johncmunson]

@BrandonCopley AWS added support for promises to the sdk back in March 2016, but the API is unconventional so I think that has caused more than it's fair share of confusion. Most popular JS libraries that I've seen that added support for promises at a later date did so by allowing you to exclude the callback. No callback? Get a promise.

With the aws-sdk, you can exclude the callback, but you also have to chain .promise() onto your method. Any function or method that returns a promise may be used with async/await.

Your stack trace doesn't appear to be related to async/await, but rather trying to push an element into an array that doesn't exist.

[BrandonCopley]

It looks like when you have .promise() chained with a function wrapped by Bluebird this bug occurs. That’s what I’m seeing.

Brandon

On Sat, Mar 30, 2019 at 7:01 PM John Munson [email protected] wrote:

@BrandonCopley https://github.com/BrandonCopley AWS added support for promises https://aws.amazon.com/blogs/developer/support-for-promises-in-the-sdk/ to the sdk back in March 2016, but the API is unconventional so I think that has caused more than it's fair share of confusion. Most popular JS libraries that I've seen that added support for promises at a later date did so by allowing you to exclude the callback. No callback? Get a promise.

With the aws-sdk, you can exclude the callback, but you also have to chain .promise() onto your method. Any function or method that returns a promise may be used with async/await.

Your stack trace doesn't appear to be related to async/await, but rather trying to push an element into an array that doesn't exist.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-js/issues/1436#issuecomment-478299698, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJShlsFTPbp1XtcmEOCkOqD_ReKWGkuks5vb_rJgaJpZM4Mt8vp .

--

[image: Giftnix]

Brandon Copley

Founder & CEO

t: 512.784.6060 <(512)%20784-6060>

e: [email protected]

[johncmunson]

@BrandonCopley yeah, idk, I'd have to see some code

[mnadeem]

app.get('/api/template',  (req, res) => {
  const { procedureCode, lobId } = req.query;
  const { membership } = req.decoded;
  
  let promises = [getS3Object(`templates/t.tpl`),
                  getS3Object(`templates/z.tpl`),
                  getS3Object(`templates/y.tpl`),
                  getS3Object('templates/x.tpl')];
  
  return Promise.all(promises)
    .then((pres) => {
      const iterable = [0, 1, 2, 3];

      for (let value of iterable) {
        if (!pres[value].statusCode) {
          res.send(pres[value]);
        }
      }
    })
    .catch((res) => {
      console.log(`Error Getting Templates: ${res}`);
    });
});

const getS3Object = key => {
  return new Promise((resolve, reject) => {
      s3.getObject({
          Key: key
      }, (err, data) => {
          if (err){
            resolve(err)
          } else {
            resolve(data.Body)
          }
      })
  })
}

[javadba]

I was also seeing that error

"TypeError: Cannot read property 'push' of undefined"

at Request.HTTP_DATA (node_modules/aws-sdk/lib/event_listeners.js:381:35)
at Request.callListeners (node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (node_modules/aws-sdk/lib/request.js:683:14)
at IncomingMessage.onReadable (node_modules/aws-sdk/lib/event_listeners.js:281:32)
at IncomingMessage.emit (events.js:189:13)
at IncomingMessage.EventEmitter.emit (domain.js:441:20)
at emitReadable_ (_stream_readable.js:535:12)
at process._tickCallback (internal/process/next_tick.js:63:19)

And no array pushes are happening in that script. The AWS SDK handling of promises seems to behave differently than the convention.

[patryk-matis]

How can I use promise with createReadStream?

        const file = await s3.getObject({
            Bucket: process.env.AWS_BUCKET_NAME,
            Key: key
        }).promise();
        
        file.createReadStream().pipe(res);

Results in: {"error":"file.createReadStream is not a function"}

[alanwilter]

Could anyone make a proper functional example please?

I'm trying with this:

Be temp.js:

const event = {
	input_file: 'uploadedfile/example.fda',
	output_s3_dir: 'processed/example_out',
};

const aws = require('aws-sdk');
const s3 = new aws.S3();
const fs = require('fs');
const path = require('path');
const { exitCode, exit } = require('process');
const execSync = require('child_process').execSync;
BUCKET = process.env.BUCKET;

const input_file = event.input_file;

function downloadImage(input_file) {
	const input_basename = path.basename(input_file);
	const docker_input_file = path.join('/tmp/', `${input_basename}`);
	const params = {
		Bucket: BUCKET,
		Key: input_file,
	};
	return new Promise((resolve, reject) => {
		const s3Stream = s3.getObject(params).createReadStream();
		const fileStream = fs.createWriteStream(docker_input_file);
		s3Stream.on('error', reject);
		fileStream.on('error', reject);
		fileStream.on('close', () => {
			resolve(docker_input_file);
		});
		s3Stream.pipe(fileStream);
	});
}

// How to return docker_input_file string value from downloadImage()?
downloadImage(input_file);
docker_input_file = path.join('/tmp/', `${path.basename(input_file)}`);

try {
	if (fs.existsSync(docker_input_file)) {
		console.log('>>> File exists');
		exit(10);
	}
} catch (err) {
	console.error('ERROR', err);
	exit(20);
}

Then:

$ node temp.js
$ echo $?
0
# what???

My point is, I really need the file to exist at the lambda node /tmp/... because I call other processes on it.

[bertdida]

In my case, I have to convert the stream to Buffer then convert back to Readable.

import { Readable } from "stream";

// inside try...catch
const stream = await new Promise<Readable>((resolve, reject) => {
  const s3Stream = s3.getObject(params).createReadStream();
  const chunks: any[] = [];

  s3Stream.on("error", reject);
  s3Stream.on("data", (chunk) => chunks.push(chunk));

  s3Stream.on("end", () => {
    const stream = new Readable();
    stream.push(Buffer.concat(chunks));
    stream.push(null);
    resolve(stream);
  });
});

[umkhan65]

res.attachment(fileKey);
      const fileStream = await s3.getObject(options).createReadStream();
      fileStream.on('error', (e) => {
        //sends error res
      });
      fileStream.pipe(res);

Won't this work? This works for me but once in a blue moon this gives an error "Cannot set headers after they are sent to the client", not sure if I should wrap this in a promise

[jrdnull]

@KidA001 you forgot to chain .promise() onto s3.getObject. See below for the correction.

For anyone using async await:

async function getS3File(filename) {
  const params = {
    Bucket: 'some-bucket-name',
    Key: filename,
    Body: fileIO,
  };

  const response = await s3.getObject(params, (err) => {
    if (err) {
      // handle errors
    }
  }).promise();

  return response.Body.toString(); // your file as a string
}

Don't mix callbacks with promise(), causes issue see https://github.com/aws/aws-sdk-js/issues/1628#issuecomment-366252338 you'll end up with partial response

Instead something like

async function getS3File(filename) {
  const params = {
    Bucket: 'some-bucket-name',
    Key: filename,
  };

  try {
    const response = await s3.getObject({
      Bucket: 'some-bucket-name',
      Key: filename,
    }).promise();
  } catch (err) {
      // handle errors
  }

  return response.Body.toString(); // your file as a string
}