gm
gm copied to clipboard
aheckmann
Getting "toBuffer() Stream yields empty buffer" error for append function
Here is my sample code var gm = require('gm').subClass({ imageMagick: true }); // Enable ImageMagick integration.
gm(imgSrc1).append(imgSrc2, true).setFormat('jpeg').toBuffer(function(err, buffer) { if (err) { next(err); } else { next(null, buffer); } });
Env: AWS Lambda Memory : 1024MB (To make sure I'm not running out of memory while testing)
The same code works on my EBS instance.
Let me know if image appending can be achieved in AWS lambda.
@Siddeshgad Have you been able to get this to work? I'm having the same issue.
@amitaymolko Not yet. I had a work around for this problem. I am processing images with this function on my server rather than automating it using AWS Lambda service.
This problem was only occurring on AWS Lambda instances. I was able to execute the same code on my local machine as well as on server.
I was able to get it to work like this:
var image = gm(body, pathname)
image
.setFormat(mime.extension(contentType))
.resize(width, height, '^')
.gravity('Center')
.crop(width, height)
.stream(function(err, stdout, stderr) {
if(err) reject(err)
var chunks = [];
stdout.on('data', function (chunk) {
chunks.push(chunk);
});
stdout.on('end', function () {
var image = Buffer.concat(chunks);
var s3_options = {
Bucket: S3_BUCKET,
Key: bucketkey,
Body: image,
ACL: 'public-read',
ContentType: contentType,
ContentLength: image.length
}
S3.upload(s3_options, (err, data) => {
if(err) {
reject(err)
}
var image_url = `${STORAGE_BASE_URL}/${bucketkey}`
resolve({original_url: url, image_url: image_url, size})
})
});
stderr.on('data',function(data){
console.log(`stderr ${size} data:`, data);
})
})
As you can see this also includes the S3 upload code. Hope this can help you.
I am getting this error locally now, on OSX. It suddenly started happening and I have no idea why. Digging in...
Ok, so my issue is that there was an error in the processing (for me, image path to a composited image was incorrect), however the toBuffer method will always return with this "empty buffer" message rather than actually showing you the real error you need to fix.
To improve on this, I used some of @amitaymolko's great code above to create a better, promise-based implementation of the toBuffer method that actually returns errors correctly if there are any:
function gmToBuffer (data) {
return new Promise((resolve, reject) => {
data.stream((err, stdout, stderr) => {
if (err) { return reject(err) }
const chunks = []
stdout.on('data', (chunk) => { chunks.push(chunk) })
// these are 'once' because they can and do fire multiple times for multiple errors,
// but this is a promise so you'll have to deal with them one at a time
stdout.once('end', () => { resolve(Buffer.concat(chunks)) })
stderr.once('data', (data) => { reject(String(data)) })
})
})
}
If any maintainers of this project are checking this out, I would be happy to PR this into the core but as a callback-based version, just let me know if you want me to!
I got the same error on heroku too.
However, toBuffer works fine under cedar-14 (ubuntu 14.04) + https://github.com/ello/heroku-buildpack-imagemagick (6.9.5-10) but return error after upgrade to heroku-16 (ubuntu 16.04)
Still don't know the root cause of this issue, so we cannot use toBuffer under new version?
@Ferrari -- I would use the version in my comment above to be sure that you are getting accurate error messages at least until it the library is patched (any maintainers listening in? I'm happy to submit a patch as long as someone gives me the go ahead here)
@jescalan What's the best way to use your gmToBuffer function? I'm not sure how to use it... what should I pass as the data attribute?
Hey @thomasjonas -- you can pass the results of a gm call directly into it 😁. So for example:
const data = gm(srcStream, 'output.jpg').resize(500)
gmToBuffer(data).then(console.log)
Thanks so much @jescalan . You saved the day :smile_cat:
Just in case anybody needs to fetch from a url, process and convert to base64, this is how I finally did it.
const data = gm(request(url)).borderColor('#000').border(1,1).resize(500,500)
gmToBuffer(data).then(function(buffer) {
let dataPrefix = `data:image/png;base64,`
let data = dataPrefix + buffer.toString('base64')
return callback(null, data)
})
.catch(function(err){
console.log(err)
return callback(err)
})
Sidenote:
AWS Lambda was crashing because I had
const gm = require('gm')
instead of
const gm = require('gm').subClass({imageMagick: true})
@jescalan 's code works for me too on Google Cloud with CentOS.
The only reasonable explanation I can think of is that the toBuffer function resolves to some underlying C code that has compatibility issues - building the buffer from the stream goes around that problem.
Hi @jescalan I am having a problem with windows. I have finished my app and it works perfectly if I run it from cmd.exe with "node myapp.js".
The problem though is that I need to put it in a scheduled task so it can be run every x hours. When the scheduler run my task I keep getting the same error. Stream yields empty buffer.
So I use your function I got a different one.
Invalid Parameter -
Any ideas?.
var fs = require('fs')
, gm = require('gm').subClass({imageMagick: true});
let fp = `${__dirname}/logo.png`
let new_fp = `${__dirname}/logo_new.png`
//resize and remove EXIF profile data
let data = gm(fp)
.resize(240, 240)
.noProfile()
.write(new_fp, function (err) {
if (!err){
console.log('done')
}else{
console.log(err);
}
})
function gmToBuffer (data) {
return new Promise((resolve, reject) => {
data.stream((err, stdout, stderr) => {
if (err) { return reject(err) }
const chunks = []
stdout.on('data', (chunk) => { chunks.push(chunk) })
// these are 'once' because they can and do fire multiple times for multiple errors,
// but this is a promise so you'll have to deal with them one at a time
stdout.once('end', () => { resolve(Buffer.concat(chunks)) })
stderr.once('data', (data) => { reject(String(data)) })
})
})
}
gmToBuffer(data).then(console.log).catch(e => console.log(e))
Might be because you're calling write on your original gm process call? It's also possible you have an error in your path and there is genuinely no data.
@jescalan I got a new error. Invalid Parameter - -resize. Looks windows task scheduler can't reference this method?.
I am lost here
I don't really think it has anything to do with windows or windows task scheduler based on that error message. I'm not entirely sure what's going wrong at this point though, nor did I write or maintain this library so my knowledge of how it works is limited. I'm sorry!
@jescalan I uninstalled imagemagick and I am getting the same error when I run it with node.
So it was not finding imagemagick when I was running with the scheduler. I don't know what to do though.
Imagemagick installation problems related to windows scheduler is way out of my range of abilities to debug. I don't use windows at all, nor do I have access to your machine or the ability to tinker with the command line. The only possible suggestion I have here is that this is not an imagemagick library, its graphicsmagick -- if you installed imagemagick instead then it would be pretty clear why it wouldn't be working.
@jescalan Finally I was able to fix it. Thanks!. (https://github.com/aheckmann/gm/issues/572#issuecomment-421423019) pointed me to the right direction.
Instead of setting the task to run
node myapp.js.
I told the task to run
cmd.exe /k cd "C:\my\path" & node myapp.js
I got this issue on Heroku when using heroku-18 stack together with ello/heroku-buildpack-imagemagick.
Removing the buildpack solved the issue for me. Seems like heroku-18 comes with imagemagick installed already. You can check with this command:
$ heroku run convert --version
Also getting this on Heroku 18 now, with or without additional buildpacks.
This buildpack worked for me: https://github.com/bogini/heroku-buildpack-graphicsmagick
I'm still getting the same error when trying to use streams as specified in the docs or even when using the code suggested above.
I already tried the following with no luck:
- with http options tweaks
- with Imagemagick
- with Graphicsmagick
- with an additional PassThrough stream used as a writeStream as the docs suggests.
This is the error I'm getting:
[2019-01-30T14:35:48+10:00] (2019/01/30/[$LATEST]2c3e75a349ac41c1ad821b37b9f3a9ee) 2019-01-30T04:35:48.960Z 389a0660-49f6-4cd8-a9f4-567d783bba56 {"errorMessage":"gm convert: Read error at scanline 7424; got 3792644 bytes, expected 9216000. (TIFFFillStrip).\n"}
Doesn't seem to be an issue with AWS SDK since if I just create a download steam with S3.getObject.createReadStream and pass it to S3.upload method it works perfectly. Meaning even with a huge file of 3gb, simply copying from/to s3 works without any issues.
The following code works with small images but it fails with a image of 1.5gb when deployed and running from AWS.
import {Handler} from "aws-lambda";
import {Helper} from "../lib/Helper";
import * as AWS from "aws-sdk";
let https = require("https");
let agent = new https.Agent({
rejectUnauthorized: true,
maxSockets: 1000
});
AWS.config.update({
"httpOptions": {
timeout: 600000,
connectTimeout: 600000,
logger: console,
agent: agent
}
});
export const handler: Handler = async (event) => {
process.env["PATH"] = process.env["PATH"] + ":" + process.env["LAMBDA_TASK_ROOT"];
const {bucket, object} = event.Records[0].s3;
let total = 0;
const file = Helper
.getS3Client()
.getObject({Bucket: bucket.name, Key: object.key})
.createReadStream()
.on("data", (data) => {
total += data.toString().length;
// console.log(`s3 stream read ${data.byteLength} bytes @ ${Date.now()}`);
})
.on("error", (err) => {
console.log(`s3 stream <${object.key}>: ${err}`);
})
.on("finish", () => {
console.log(`s3 stream finished ${object.key}`);
})
.on("end", () => {
console.log(`s3 stream successfully downloaded ${object.key}, total ${Helper.bytesToSize(total)}`);
});
const gm = require("gm").subClass({imageMagick: false});
const img = await new Promise((resolve, reject) => {
gm(file, Helper.getFilenameFromS3ObjectKey(object.key))
.resize(
Helper.THUMBNAIL_WIDTH,
Helper.THUMBNAIL_HEIGHT
)
.quality(Helper.THUMBNAIL_QUALITY)
.stream("jpg", (err, stdout, stderr) => {
if (err) {
return reject(err);
}
const chunks = [];
stdout.on("data", (chunk) => {
chunks.push(chunk);
});
stdout.once("end", () => {
resolve(Buffer.concat(chunks));
});
stderr.once("data", (data) => {
reject(String(data));
});
});
});
const promise = new Promise((resolve, reject) => {
Helper
.getS3Client()
.upload({
Body: img,
Bucket: bucket.name,
Key: Helper.getThumbFilename(object.key)
})
.promise()
.then(() => {
agent.destroy();
resolve();
})
.catch((err) => {
console.log(err);
});
});
await promise;
};
Interestingly enough It does work fine, even with a 1.5gb image, when testing locally with lamci/lambda:
docker run -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -v "$PWD/dist":/var/task lambci/lambda:nodejs8.10 index.handler '{ "Records": [{ "s3": { "bucket": { "name": "xyz-images" }, "object": { "key": "bigfile.tiff" } } }] }'
Looks like my box is fast enough to consume the readStream created by S3.getObject.createReadStream and convert the image, whereas the box in AWS isn't and messes up the stream flow.
I'm finding that I get this error due to the initial image resolution rather than just the file size. My script can handle a 30mb file fine at 72dpi but I get the empty buffer message on a 12mb image at 400dpi. The same image uploaded at 350dpi processes just fine. I've increased the memory allocation for the Lambda function but still haven't had success with 400dpi images. I'm unfamiliar with what is behind the curtain of ImageMagick but could this be a limitation in the size of the buffer created by high-resolution images?
A little more digging reveals that there might be a cap to the buffer size in imageMagick. https://forums.aws.amazon.com/thread.jspa?threadID=174231 Not sure if increasing the Lambda timeout and memory will alleviate this, or it is a limitation of imageMagic.
We moved all our image processing to kraken.io to avoid running into the constant issues related to running imagemagick on our own server. Would highly recommend it if all you're doing is resizing/scaling/optimising images.
I'm see the ImageMagick library have a parameter -limit memory xxx and it's work fine with command line tool, but it's not working in gm library.
when I see the source about gm, I'm found the limit function in args.js
proto.limit = function limit (type, val) {
type = type.toLowerCase();
if (!~limits.indexOf(type)) {
return this;
}
return this.out("-limit", type, val);
}
Do you see, it's called this.out function, that means it's will be append the -limit parameter after to source, like thisconvert xxx.jpg -limit memory 512 and not to convert -limit memory 512 xxx.jpg, so we just need change the code to :
return this.in("-limit", type, value);
It's working now... my test picture resolution is: 20386x8401
This might be unrelated but I didn't start having this buffer error until I changed my node version in lambda from like 6 to 10. I think this might actually be an issue with bumping the node version.
EDIT: CONFIRMED. My problems went away entirely by downgrading to nodejs8.10
I'm experiencing the same issue. First it was because I was using toBuffer method. Then I changed to .stream method but it was returning an empty stream. I added console.logs at on('data', ... and on('end', ... to verify and it was only printing end.
Then I changed imageMagick from true to false:
Before:
const gm = require('gm').subClass({ imageMagick: true });
After:
const gm = require('gm').subClass({ imageMagick: false });
and it started working, but only in my local machine.
I executed the function using SAM CLI but it started uploading files with 0 size.
I uploaded code to AWS Lambda and the same is happening, it uploads files with 0 size. Only in my local machine is working.
This is my complete code:
const gm = require('gm').subClass({ imageMagick: false });
const AWS = require('aws-sdk');
const s3 = new AWS.S3();
exports.handler = async (event, context, cb) => {
const validExtensions = ['jpg', 'jpeg', 'png'];
const { bucket, object } = event.Records[0].s3;
// Where images are uploaded
const origin = 'original/';
// Where optimized images will be saved
const dest = 'thumbs/';
// Object key may have spaces or unicode non-ASCII characters. Remove prefix
const fullFileName = decodeURIComponent(object.key.replace(/\+/g, ' '))
.split('/').pop();
const [fileName, fileExt] = fullFileName.split('.');
if (!validExtensions.includes(fileExt)) {
return cb(null, `Image not processed due to .${fileExt} file extension`);
}
// Download image from S3
const s3Image = await s3.
getObject({
Bucket: bucket.name,
Key: `${origin}${fullFileName}`
})
.promise();
function gmToBuffer(data) {
return new Promise((resolve, reject) => {
data.stream((err, stdout, stderr) => {
if (err) { return reject(err) }
const chunks = []
stdout.on('data', (chunk) => { chunks.push(chunk) })
stdout.once('end', () => { resolve(Buffer.concat(chunks)) })
stderr.once('data', (data) => { reject(String(data)) })
})
})
}
function getStream(body, size, quality) {
const data = gm(body)
.resize(size)
.quality(quality);
return gmToBuffer(data);
}
// use null to optimize image without resizing
const sizes = [null, 1200, 640, 420];
// Uploades all images to S3
const uploadPromises = sizes.map(async size => {
// Optimize image with current size
const readableStream = await getStream(s3Image.Body, size, 60);
const key = size
? `${dest}${fileName}_thumb_${size}.${fileExt}`
: `${dest}${fileName}_original.${fileExt}`;
return s3.putObject({
Bucket: bucket.name,
Key: key,
Body: readableStream,
}).promise();
});
await Promise.all(uploadPromises);
cb(null, 'finished');
};
To execute in local I added to the end the function execution:
exports.handler(require('./event.json'), null, (err, res) => {
console.log(res);
});
I'm using node v10.x as runtime both in local and in AWS.
These are the outputs:
Local:
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
2019-08-09T16:48:24.654Z 52fdfc07-2182-154f-163f-5f0f9a621d72 INFO finished
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Duration: 1228.68 ms Billed Duration: 1300 ms Memory Size: 128 MB Max Memory Used: 61 MB
AWS:
START RequestId: b8bd8a36-f56b-4732-b9eb-a03fa6c5f75b Version: $LATEST
2019-08-09T16:31:32.994Z b8bd8a36-f56b-4732-b9eb-a03fa6c5f75b INFO finished
END RequestId: b8bd8a36-f56b-4732-b9eb-a03fa6c5f75b
REPORT RequestId: b8bd8a36-f56b-4732-b9eb-a03fa6c5f75b Duration: 818.42 ms Billed Duration: 900 ms Memory Size: 256 MB Max Memory Used: 101 MB
So, I do not think it is an issue related to memory, but still I'm using 256 MB of RAM and 15s as timeout.
@rtalexk
Set Node to nodejs8.10, set imageMagick=true.
Try that in AWS.
@rtalexk Also maybe dont use that guys gmToBuffer() thing. Try to use the built in GM toBuffer and callback. I know its not a promise but try eliminating all that extra fluff. And like I said above, most of all, Set Node to nodejs8.10
I changed te runtime to 8.10 and I also changed the code to not use gmToBuffer function (I don't think it affect anything because it's only a wrapper to convert .stream method to be promise-based) but still the same.
Running the function directly with node (node index.js) works as it should. But once I upload the code to AWS it starts generating images with 0 B of size. The same happens while running with SAM CLI.
It's weird because the only difference is where the code runs. Input, code and libraries (with versions) are the same. I'm using the AWS official docs as reference for this exercise: https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
Oh, I see. Something is wrong with GM. What else could I use to resize/optimize images? I was looking to imagemin with imagemin-pngquant and imagemin-mozjpeg for support to png and jpg/jpeg but it also have problems while running in AWS Lambda. I tried different workarounds but always there's a complex solution that add complex to your system.
Right now I'm looking into Jimp. It looks promising.
I implemented Jimp and everything is working.
The only inconvenient is how much it takes to optimize images:
REPORT RequestId: ... Duration: 14629.42 ms Billed Duration: 14700 ms Memory Size: 512 MB Max Memory Used: 359 MB
It was tested using the following image:
Dimensions: 2048x1365
Size: 250 KB
It generates 4 new images:
| Original | Image 1 | Image 2 | Image 3 | Image 4 | |
|---|---|---|---|---|---|
| Dimensions (px) | 2048x1365 | 2048x1365 | 1200x800 | 640x427 | 420x280 |
| Size (KB) | 250 | 176 | 66 | 25 | 12 |
I'm worried about how much it is going to cost with larger images and hundreds/thousands of requests.
If you're interested in code, it is available at https://github.com/rtalexk/lambda-image-optimization
For users stringling to make AWS Lambda & Node.js 10 & ImageMagick work
I was able to solve the issue with the following resolution : https://github.com/aheckmann/gm/issues/752#issuecomment-545397275
