cloudtrail-parquet-glue icon indicating copy to clipboard operation
cloudtrail-parquet-glue copied to clipboard

Published 20 hours ago • verified publisher icon alsmola

Recrawl Policy

Open ryanmac8 opened this issue 4 years ago • 0 comments

I've noticed that the glue crawlers are currently setup to crawl all folders each time it's ran. Shouldn't it be changed to only crawl new folders only?I have millions of files to crawl through and it takes forever. If it's only set to crawl the new folders, it should make the process much faster.

I believe if we implement the Recrawl Policy on the terraform, this would solve the issue.

ryanmac8 avatar Feb 11 '21 21:02 ryanmac8