graylog-plugin-aws icon indicating copy to clipboard operation
graylog-plugin-aws copied to clipboard

Speed up reading of CloudTrail logs from SQS

Open lennartkoopmann opened this issue 9 years ago • 1 comments
trafficstars

Reading thousands of CloudTrail logs can be very slow. There must be a better way to read everything in batches, using their cursor functionality.

lennartkoopmann avatar Nov 06 '16 21:11 lennartkoopmann

I'll also try to read from S3 in batches or multithreaded. This is where the bottleneck is now.

lennartkoopmann avatar Nov 06 '16 23:11 lennartkoopmann