Robert (Bobby) Evans
Robert (Bobby) Evans
This might be similar to #6001. This exception is caused by us trying to shut down RMM, but there is memory that is still outstanding. This could be related to...
> We should add the PsNote for `ArrayType(FloatType)` and `ArrayType(DoubleType)` I guess. But it seems like the `TypeEnum` doesn't have a constructor for nested types. In those cases we usually...
Talking to @jlowe I think we probably also want to add in an error so that if we add a psNote to a type that Spark does not support, that...
> No, I don't think it would be necessary. If the regex kernels don't have an appreciably larger footprint than any other random libcudf kernel then I don't see the...
The code to handle this is here https://github.com/NVIDIA/spark-rapids/blob/2b9fa502caac77b451e8a6dc84842a3b182592c8/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetFileFormat.scala#L304-L350 It should work. Not 100% sure what is happening that is causing it to fail.
The documentation needs to be cleaned up to make this clear. We do not support running arbitrary java/scala code on the GPU. We have some experimental support that can [transpile...
@SidWeng is it okay if I hijack this and turn it into a research spike to see what it might take to add any dataset support at all?
> Try to understand more detail, do you mean the byte code analysis should be done in GpuOverrides.wrapPlan()? From my current knowledge, GpuOverrides.wrapPlan() maps SparkPlan into SparkPlanMeta which wraps ExecRule,...
@SidWeng I have started to take a look at this and it is very complicated. It is likely to involve a lot of byte code analysis to have any hope...