Error while downloading huge log files

[ghost]

2019-07-24 07:35:05,150 :: INFO :: Getting logs from 2019-07-23
2019-07-24 07:35:05,352 :: INFO :: Downloading file /home/postgres/logs/vendor-live//error/postgresql.log.2019-07-23-00
2019-07-24 07:35:05,996 :: INFO :: Log truncated, retrying portion with NumberOfLines = 948
2019-07-24 07:35:08,924 :: INFO :: Log truncated, retrying portion with NumberOfLines = 852
2019-07-24 07:36:25,812 :: ERROR :: An error occurred (Throttling) when calling the DownloadDBLogFilePortion operation (reached max retries: 4): Rate exceeded

[Lngramos]

I've just run into the same issue and increasing Boto's default number of retry attempts as per https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html seemed to help.

You can also specify these values either as local config file or env vars: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html#cli-usage-retries-configure-envvar