Robert (Bobby) Evans

Results 257 comments of Robert (Bobby) Evans

The changes look good to me. I ran some tests. ``` val dfs = (0 until 9).map(i => math.pow(10, i).toLong).map{ rows => spark.range(rows).selectExpr(s"rand($rows) as rn").selectExpr("percentile(rn, array(0.1, 0.2, 0.3, 0.4, 0.5,...

Could we optionally include a lit of metrics so that it matches `NvtxWithMetrics` and then we just totally delete `NvtxWithMetrics`.

This is a performance optimization in Spark that is only supposed to happen when the value would not be impacted by potential floating point issues. https://github.com/apache/spark/blob/9bb358b51e30b5041c0cd20e27cf995aca5ed4c7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L2110-L2142 So if the precision...

I am remembering more now. Converting a double to a decimal value has problems because they do it by going from a double to a string to a decimal. This...

So java does odd things when interpreting floating point values compared to the rest of the world. They try to fix the problem that some decimal values cannot be represented...

But before we do any of that I want to be sure that we know what the original input long was before the divide happened and what the double was...

@thirtiseven yes that is one possibility, but again it is almost match. That is up to management about how close is good enough.

I should add that an empty struct results in a different error. ``` Seq("""{"a":1,"b":"","c":{}}""").toDF("json").repartition(1).selectExpr("from_json(json, 'a int, b string, c struct')").show() ``` ``` Caused by: java.lang.NullPointerException at ai.rapids.cudf.Table.gatherJSONColumns(Table.java:1105) at ai.rapids.cudf.Table.gatherJSONColumns(Table.java:1225) at...

This could be a huge win for memory on cases like https://github.com/NVIDIA/spark-rapids/issues/10561 where we insert in a lot of null columns as place holders knowing that they will never be...

I filed https://github.com/rapidsai/cudf/issues/15308 for this in CUDF. We will see what happens.