spark-rapids icon indicating copy to clipboard operation
spark-rapids copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

[WIP] initial PR for the framework resuing Vanila Spark's UTs (borrow from …

Open binmahone opened this issue 10 months ago • 5 comments

…Gluten)

This PR close https://github.com/NVIDIA/spark-rapids/issues/10745

This PR adds a framework to enabling Vanilla Spark UTs by:

  1. Some utility Traits, most importantly RapidsTestsTrait and RapidsSQLTestsTrait for rapids test suite to extend, these traits: injects rapids backboned spark session to the base class, as well as allow testcase level exclusion/inclusion.
  2. A centralized control center called RapidsTestSettings, which decides what suites/testcases are included/excluded.
  3. A first batch of 8 suites are enabled, listed at https://github.com/NVIDIA/spark-rapids/pull/10743/files#diff-4be8dbebdd70379fcc56bae360c6997a9a2b6e66ce48b229c7ff289ae02a5222R41

Most of codes are copied from Apache Gluten, which also follows Apache license. To respect their work I kept their origin Apache License header in the source code file.

Currently we only focus on spark 3.3.1.

Check out more details at our internal design doc.

binmahone avatar Apr 25 '24 12:04 binmahone

build

binmahone avatar Apr 28 '24 02:04 binmahone

build

binmahone avatar Apr 28 '24 06:04 binmahone

build

binmahone avatar Apr 28 '24 06:04 binmahone

build

binmahone avatar Apr 28 '24 09:04 binmahone

build

binmahone avatar Apr 28 '24 15:04 binmahone

build

binmahone avatar Apr 30 '24 09:04 binmahone

Hi @GaryShen2008 @jlowe the PR has passed all CI checks, can you please help to review?

binmahone avatar May 01 '24 08:05 binmahone

build

binmahone avatar May 07 '24 05:05 binmahone

build

binmahone avatar May 07 '24 05:05 binmahone

build

binmahone avatar May 08 '24 08:05 binmahone

build

binmahone avatar May 08 '24 13:05 binmahone

build

binmahone avatar May 09 '24 03:05 binmahone