cloudtrail-parquet-glue

cloudtrail-parquet-glue copied to clipboard

Solution does not scale for large existing AWS accounts.

3

I have a large existing AWS account with a few years of CloudTrail log data already in my source S3 bucket. When deploying this solution and manually running the first...

BigDataDaddy

Fixed partitioning issue during raw to parquet

1

Changed mappings in glue_etl.py to tie the Glue-given "partition_[0-6]" names to awslogs, account, region, etc. This should fix the errors being referenced in Issue 1.

andrew-kline

Initialization help

7

In order to use this project, is it sufficient to create the 3 new S3 buckets and then fill in the values in the `variables.tf` file and then run `terraform...

0xdabbad00

I've tried setting this up to compare it against an existing non-Glue-based solution, and am running into a couple of issues I'm hoping someone can help with. Disclaimer: I have...

bcenker

I've noticed that the glue crawlers are currently setup to crawl all folders each time it's ran. Shouldn't it be changed to only crawl new folders only?I have millions of...

ryanmac8

To be really useful in a production account sense this module needs to support some form of tagging on the AWS resources created. I suggest something like a list of...

BigDataDaddy

Question - Data retention and Bucket Lifecycle policies

1

Thanks for sharing this project and the extensive blog post behind it. I've got a few questions - While I've manually setup athena to make adhoc queries against cloudtrail buckets...

dlethin

Fixed extra / missing characters

gescarra