Josh Rosen

I could just add a `"provided"` dependency on `spark-core`, `spark-streaming`, `spark-streaming-kafka`, etc, but this results in the provided version being used in the `test` classpath, so we have to do...

Hi @lehnerm, Just to help me narrow down potential causes, could you let me know which Spark Redshift version you're using and which AWS region(s) are hosting your Spark driver,...

@lehnerm, I think that I may have some contacts on the S3 and Redshift teams, so I'll forward this thread to them to see if they have any insights. Ping...

Taking another look at the [Amazon S3 Data Consistency Model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel) docs (emphasis mine): > Amazon S3 achieves high availability by replicating data across multiple servers within Amazon's data centers. If...

In addition, according to the documentation for [Redshift manifest files](https://docs.aws.amazon.com/redshift/latest/dg/managing-data-consistency.html#using-a-manifest-file), emphasis mine: > You can explicitly specify which files to load by using a manifest file. When you use a...

@lehnerm, a couple of other questions that I just thought of: - Have you configured and Hadoop or Spark OutputCommitter settings to be different than their defaults? - Are you...

Based on a more careful reading of the announcement about the availability of strong read-after-write consistency in all regions (https://forums.aws.amazon.com/ann.jspa?annID=3112), it sounds like this might actually be saying that each...

I guess we need another layer of `try-finally` in the `finally` block to log and ignore exceptions thrown there so that we don't mask the original cause / exception.

Not a solution to your problem necessarily but I think that that the changes in #157 should have addressed the silent exception loss issue that made this ticket harder to...

Another way to avoid this problem would be to have `spark-redshift` truncate the existing table then load new rows rather than dropping and re-creating the table. The current behavior in...