initialization-actions icon indicating copy to clipboard operation
initialization-actions copied to clipboard
verified publisher icon GoogleCloudDataproc

[spark-rapids] Update spark rapids version to 24.04.0

Open SurajAralihalli opened this issue 1 year ago • 2 comments

This PR updates

  1. spark-rapids.sh init script with the latest 24.02.0 (to-date) rapids-4-spark version.
  2. Changes default driver version to 550.54.15 and cuda 12.4.1
  3. Uses apt-get --allow-releaseinfo-change update to unblock downstream applications until dataproc platform fixes apt-get update issues in Debian 10 and Ubuntu linux distros.

signed-off-by: Suraj Aralihalli [email protected]

SurajAralihalli avatar Apr 23 '24 17:04 SurajAralihalli

FYI @viadea @jayadeep-jayaraman

SurajAralihalli avatar Apr 24 '24 23:04 SurajAralihalli

/gcbrun

jayadeep-jayaraman avatar May 17 '24 08:05 jayadeep-jayaraman

/gcbrun

jayadeep-jayaraman avatar May 20 '24 03:05 jayadeep-jayaraman 

+ echo 'Error: Secure Boot is enabled. Please disable Secure Boot while creating the cluster.'
Error: Secure Boot is enabled. Please disable Secure Boot while creating the cluster.
+ exit 1

We need to disable secure-boot for 2.2 ubuntu

jayadeep-jayaraman avatar May 20 '24 05:05 jayadeep-jayaraman 

+ echo 'Error: Secure Boot is enabled. Please disable Secure Boot while creating the cluster.'
Error: Secure Boot is enabled. Please disable Secure Boot while creating the cluster.
+ exit 1

We need to disable secure-boot for 2.2 ubuntu

@jayadeep-jayaraman Is there any action item needed in this PR? I think this error just tells us we need to disable secure boot for 2.1 and 2.2 dataproc in your CICD pipeline.

viadea avatar May 20 '24 16:05 viadea

We need to disable secure-boot for 2.2 ubuntu

Yes we need to disable secure boot for Dataproc 2.1 and 2.2 to install the drivers. A note to disable secure boot is added in the Create a Dataproc cluster accelerated by gpus docs for users.

SurajAralihalli avatar May 20 '24 16:05 SurajAralihalli

The test is failing for 2.2 ubuntu and we should bypass the test for this image version in the PR

jayadeep-jayaraman avatar May 20 '24 17:05 jayadeep-jayaraman

The test is failing for 2.2 ubuntu and we should bypass the test for this image version in the PR

@jayadeep-jayaraman i do not think we can run the CICD test to confirm. Do you want to fix this in this PR or you want us to fix this?

viadea avatar May 20 '24 17:05 viadea

The test is failing for 2.2 ubuntu and we should bypass the test for this image version in the PR

Secure boot needs to be disabled for ubuntu 22. Would you recommend bypass this check however I think this check will be useful to the users to identify the issue when secure boot is enabled.

SurajAralihalli avatar May 20 '24 18:05 SurajAralihalli

@jayadeep-jayaraman @viadea @sameerz I've updated the PR to run the skip the tests on 2.0 and 2.1 (due to secure boot issue). I'm confident that the tests would succeed if we find a way to disable the secure boot in CI. This means the tests would run only on Debian 10 (2.0) as we are in the processing of dropping support for Ubuntu 18. To unblock the PR from being merged asap I've included --allow-releaseinfo-change.

SurajAralihalli avatar May 21 '24 21:05 SurajAralihalli

@jayadeep-jayaraman @viadea @sameerz I've updated the PR to run the skip the tests on 2.0 and 2.1 (due to secure boot issue). I'm confident that the tests would succeed if we find a way to disable the secure boot in CI. This means the tests would run only on Debian 10 (2.0) as we are in the processing of dropping support for Ubuntu 18. To unblock the PR from being merged asap I've included --allow-releaseinfo-change.

LGTM. @jayadeep-jayaraman shall we merge it after test is passed?

viadea avatar May 21 '24 21:05 viadea

/gcbrun

jayadeep-jayaraman avatar May 22 '24 11:05 jayadeep-jayaraman