Uncaught Error while sending?

[conniden]

Great script! I have it working 99% of the time. Unfortunately, I'm getting occasional errors that result in the dropping of a logfile. They seem to occur most often when traffic is heaviest. Here's a sample error from the CloudWatch Stream:

2015-12-31T04:07:34.174Z    92fd45fa-af73-11e5-9e41-c1b36be05f9c    TypeError: Uncaught, unspecified "error" event.
at TypeError (<anonymous>)
at Transform.emit (events.js:74:15)
at Transform.onerror (/var/task/node_modules/csv-streamify/node_modules/through2/node_modules/readable-stream/lib/_stream_readable.js:604:12)
at Transform.emit (events.js:95:17)
at onwriteError (_stream_writable.js:239:10)
at onwrite (_stream_writable.js:257:5)
at WritableState.onwrite (_stream_writable.js:97:5)
at afterTransform (_stream_transform.js:99:5)
at TransformState.afterTransform (_stream_transform.js:74:12)
at Transform.parse_s3_log [as _transform] (/var/task/elb2loggly.js:199:7)

Unfortunately, I'm not familiar enough with stream modules on to pin down the problem. I suspect the Loggly receiver is throwing an error. Can you point me in a good direction to try to track this down? Alternatively, is there some error handling that could be added so the file isn't just skipped? Thanks!

[cboscolo]

@conniden I think your suspicion of the problem is probably correct. Unfortunately, debugging deployed Lambda functions isn't easy. At the very least, the script should log which S3 object it is working on along with its size, and whether or not it completed successfully.

As for making the file processing a little more robust, one thought would be to use a queue like SQS. Enqueue the S3 objects that need to be worked on, and only dequeue them if they processed completely. This solution isn't perfect as you might end up with duplicate log entries if it fails half way through the file, and get's reprocessed successfully.

I don't have much time to add this, but if you want to take a stab at it and submit a pull request we'd appreciate the enhancement!

[conniden]

Thanks @cboscolo. I made some changes to make the parsing more flexible (for my purposes), which I'll submit soon. I'll take a look at some error catching for the .post and/or some sort of queueing capability. Can you share some of the thinking behind using .pipe's to stream each line, rather than converting the whole file, then sending? I'm assuming it makes the function more scalable (saving memory). Not questioning your use of them (it looks pretty cool), I'd just like to understand their utility better (since I'd need to muck around in there if I'm gonna touch the sending code).

[cboscolo]

@conniden, please feel free to question my decisions. When I initially wrote this script I knew very little about Lambda, or Node.js for that matter. My concern at the time was that If the script held the whole file in memory then it would potentially hit the Lambda memory limit as I wasn't sure what the max size of the S3 log objects could be. Are you thinking that doing the discrete steps in memory makes error handling a little more robust?

[ctrlrsf]

IMHO handling the log file as a stream, one line or a chunk of lines at a time, is ideal. This would allow processing of arbitrarily sized log files without concern for memory consumption.

[conniden]

I don't disagree with using the streaming, it's just that it is complicating my ability to debug what's happening. :-) I'm still getting mysterious dropouts of data, though no errors are reported. I've opened a case with Loggly to see if they can shed any light.

Another issue has raised its head though. The bulk endpoint will only accept "batches" up to 5mb - which (pending confirmation from Loggly) appears to be equivalent to "files". My logfiles are currently averaging around 3mb every five minutes (x2 from the ELB). These will surely grow to over 5mb within the next few months (our traffic is seasonal).

I suspect my mysterious dropouts might be due to this filesize limit, but we'll see what Loggly support has to say.

[ctrlrsf]

@conniden To help with debugging this Lambda app in the past, I've added print statements during certain key parts to see where the errors were getting thrown or which log entries were not processed properly. Some things I logged were certain request parameters I thought could be triggering it, or length of the log line.

Also, I can confirm Loggly does have a 5MB limit. To handle this for other log importers, I store logs that would be sent to Loggly in a buffer and flush it before it gets to 5MB.

Here's how I do this for another project (written in Go though): https://github.com/zumba/cloudfront-loggly-uploader/blob/master/loggly/loggly.go