TypeError: 'JavaPackage' object is not callable, when spark-commit xgboost_example

[00-Luoshu]

• env:

  • python:3.7
  • java:1.8
  • spark:3.1.2

• The doc and code comes from https://github.com/intel-analytics/BigDL/tree/main/python/dllib/examples/nnframes/xgboost https://github.com/intel-analytics/BigDL/blob/main/python/dllib/examples/nnframes/xgboost/xgboost_example.py

• My spark-submit command:

    /home/intel/spark-3.1.2-bin-hadoop2.7/bin/spark-submit \
    --master "local[4]" \
    --driver-memory 2g \
    --executor-memory 2g \
    local:///home/zhangyu/BigDL/python/dllib/examples/nnframes/xgboost/xgboost_example.py --file-path data/sample.csv 

• And error message

Traceback (most recent call last):
  File "/home/zhangyu/BigDL/python/dllib/examples/nnframes/xgboost/xgboost_example.py", line 95, in <module>
    Processdata(datapath, option.demo)
  File "/home/zhangyu/BigDL/python/dllib/examples/nnframes/xgboost/xgboost_example.py", line 47, in Processdata
    sc = init_nncontext(sparkConf)
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/nncontext.py", line 536, in init_nncontext
    sc = init_internal_nncontext(conf, spark_log_level, redirect_spark_log)
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/nncontext.py", line 665, in init_internal_nncontext
    redire_spark_logs()
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/utils/common.py", line 475, in redire_spark_logs
    callBigDlFunc(bigdl_type, "redirectSparkLogs", log_path)
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/utils/common.py", line 622, in callBigDlFunc
    for jinvoker in JavaCreator.instance(bigdl_type, gateway).value:
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/utils/common.py", line 58, in instance
    cls._instance = cls(bigdl_type, *args)
  File "/home/zhangyu/.conda/envs/zhangyu/lib/python3.7/site-packages/bigdl/dllib/utils/common.py", line 107, in __init__
    self.value.append(getattr(jclass, "ofFloat")())
TypeError: 'JavaPackage' object is not callable

And I didnot downloaded Analytics Zoo zip package, just use spark-submit How can I fix this?

[sgwhat]

Reason:

Lack of BigDL jar files in your script, JavaPackage happens because python cannot find the java class.

Solutions:

Use spark-submit

1. Download bigdl dllib jars file from BigDL DLLib Jars.
2. Submit following the script below:

/home/intel/spark-3.1.2-bin-hadoop2.7/bin/spark-submit \
  --master "local[4]" \
  --driver-memory 2g \
  --executor-memory 2g \
  --jars /path/to/bigdl-dllib-spark_${SPARK_VERSION}-jar-with-dependencies.jar \
  local:///home/zhangyu/BigDL/python/dllib/examples/nnframes/xgboost/xgboost_example.py --file-path data/Boston_Housing.csv

Use spark-submit-with-bigdl

Submit following the script below:

/home/zhangyu/BigDL/scripts/spark-submit-with-bigdl \
   --master ${MASTER} \
   --driver-memory 2g \
   --executor-memory 2g \
   local:///home/zhangyu/BigDL/python/dllib/examples/nnframes/xgboost/xgboost_example.py --file-path data/Boston_Housing.csv 

NOTE:

• We have migrated Analytics-zoo to BigDL, you only need to download BigDL packages (not Analytics-zoo) if needed.
• It's recommended for you to use Boston_Housing.csv instead of sample.csv.