incubator-gluten icon indicating copy to clipboard operation
incubator-gluten copied to clipboard

[VL] Results are mismatch with vanilla Spark when get_json_object({"dScore":0.0215434648799772}, "$.dScore")

Open kecookier opened this issue 1 year ago • 5 comments

Backend

VL (Velox)

Bug description

The following SQL might lead to wrong results, but it's not yet certain if there are other factors involved. I will try to reproduce later.

select get_json_object(factor_context, '$.factor_trust_model_score') as model_score
                FROM mart_finrisk.dwd_risk_antifraud_approve_scene_log_inc_d;

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

|gluten_model_score |  vanilla_model_score
+--------------------+-----------------------
| 0.0215435          | 0.0215434648799772
| 0.0128806          | 0.012880573063864434
| 0.0114058          | 0.011405845787963208
| 0.00863517         | 0.008635174896913173

kecookier avatar Mar 12 '24 04:03 kecookier

The following unit test case can reproduce the issue. I'm sure that got wrong value while parsing double in function SIMDGetJsonObjectFunction::extractStringResult(). More info about UT can refer to this commit https://github.com/kecookier/velox/commit/f87115d074772f3c00d326cd607742125b101beb#diff-fdbb1f97f88c92ea26b933c047486a61efcf71cc9cd1995f77d09fd3c7578d7aR39

  EXPECT_EQ(
      "0.0215434648799772",
      getJsonObject(R"({"dScore":0.0215434648799772})", "$.dScore"));

---------------------------------------------------------------
[zk] dv:0.0215435 numberResult:0.0215435
Expected equality of these values:
  "0.0215434648799772"
    Which is: 0x53d043d
  getJsonObject(R"({"dScore":0.0215434648799772})", "$.dScore")
    Which is: ("0.0215435")

Hi @PHILO-HE , reviewing the commit history, I believe you have more expertise in this section. Would you be willing to assist in resolving this issue, please?

kecookier avatar Mar 13 '24 03:03 kecookier

Thanks for reporting this issue! I will take a look.

PHILO-HE avatar Mar 14 '24 04:03 PHILO-HE

Hi @kecookier, it looks the below small patch can fix this issue. Please help verify it. Thanks!

https://github.com/PHILO-HE/velox/commit/a1e9be059ceb03ca4e1c68145e516a9bb07ed956

PHILO-HE avatar Mar 14 '24 08:03 PHILO-HE

Hi @kecookier, it looks the below small patch can fix this issue. Please help verify it. Thanks!

PHILO-HE/velox@a1e9be0

@PHILO-HE Thanks for your help, I'll try it out later.

kecookier avatar Mar 14 '24 09:03 kecookier

Hi @PHILO-HE , I have tested it, and that patch can fix the bug.

Hi @kecookier, it looks the below small patch can fix this issue. Please help verify it. Thanks! PHILO-HE/velox@a1e9be0

@PHILO-HE Thanks for your help, I'll try it out later.

kecookier avatar Mar 26 '24 09:03 kecookier