gatk
gatk copied to clipboard
broadinstitute
Update to Spark 3.0 and Java 11 (or 17)
Hey, just want to know that does GATK 4.1.8.1 support spark 3.0.1 ? If it doesn't support, do you guys have timeline to support Spark 3.x ? One more suggestion, if you guys mention version of spark it support for each version would be helpful. Thank you
@Abby3017 We are hoping to add support for Spark 3.x within the next few months. The currently-supported Spark version as of GATK 4.1.9.0 is 2.4.5. I agree that it would be helpful to make the supported Spark version easier to discover -- I've opened a ticket to modify GATK to print the Spark version on startup: https://github.com/broadinstitute/gatk/issues/7027
@lbergelson There is currently an intermittent segfault in the GKL on travis in Java 11 that might pose an obstacle for this ticket:
https://github.com/broadinstitute/gatk/issues/6649 https://github.com/Intel-HLS/GKL/issues/133
Yes, this is not ideal. It's weird that this just started. I wonder what's changed in the environment.
Hello,
Wondering what the status of this is? It seems like the java 11 issue is closed, so that blocker is gone correct? We would really like spark 3.0.1 + for compatibility with other tools, thanks.
@lbergelson Could you comment on what's holding this migration up when you get a chance?
What's holding us up is me being very and busy with other things. I don't think there are any technical problems that we know about. The newest disq releases support 3+. We just need to upgrade and test to find out what we broke.
Upgrading to Spark 3.3 (switches to log4j 2.x) when it's released would be ideal since it would solve the security issue mentioned here
@cmnbroad is currently working on this -- we're hoping to update GATK to Java 17, if possible
