#349 bug fix for convert parquet to dwrf memory double free error

[KevinyhZou]

What changes were proposed in this pull request?

bug fix for convert parquet to dwrf memory double free error as describe in #349

How was this patch tested?

manual tests

1. run tpch_convert_parquet_dwrf.sh it cause the error describe in #349
2. modfiy the jni_common.h as in pr, and re-run the script, no errors, and the dwrf file was produced.

[github-actions[bot]]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[Gluten-${ISSUES_ID}] ${detailed message}

See also:

• Other pull requests

[JkSelf]

@KevinyhZou Is this error related to the size of the dataset?

[KevinyhZou]

@KevinyhZou Is this error related to the size of the dataset?

the original parquet file's size is about 11KB, not big, do you mean the small data size may cause this error ? @JkSelf

[JkSelf]

@KevinyhZou Is this error related to the size of the dataset?

the original parquet file's size is about 11KB, not big, do you mean the small data size may cause this error ? @JkSelf

I re-run the tpch_convert_parquet_dwrf.sh script and can not reproduce this issue. I tested the 448M lineitem file and 3K nation file. Here is my test code

[KevinyhZou]

that is strange. here below is my test scala code

and sometimes it can pass, and sometimes not . and here is tpch_convert_parquet_dwrf.sh, I have modify some parameters as below

@JkSelf does these modify can cause the error?

[JkSelf]

@KevinyhZou It seems this issue is not related with the configurations. I use your configuration and still can not reproduce this issue.

[jinchengchenghh]

Since we cannot regenerate this case, and now we support parquet, you can try parquet format. If new information, you can reopen again.

[jinchengchenghh]

I face the same issues, and use this solution, thanks for your contribution. https://github.com/oap-project/gluten/pull/604/commits/2b0f6e422dc503d022aefdb03ba2980ae224c445

[jinchengchenghh]

Can you resolve the conflict? Or just solve it in my PR https://github.com/oap-project/gluten/pull/604?

[KevinyhZou]

I face the same issues, and use this solution, thanks for your contribution. 2b0f6e4

OK

[github-actions[bot]]

https://github.com/oap-project/gluten/issues/349