dding3

The error is: Failed to execute goal on project bigdl-orca-spark_2.1.1: Could not resolve dependencies for project com.intel.analytics.bigdl:bigdl-orca-spark_2.1.1:jar:0.14.0-SNAPSHOT: The following artifacts could not be resolved: com.intel.analytics.zoo:zoo-core-tfnet-mac:jar:0.14.0-SNAPSHOT, com.intel.analytics.zoo:zoo-core-openvino-java-mac:jar:0.14.0-SNAPSHOT, com.intel.analytics.zoo:zoo-core-mkl-mac:jar:0.14.0-SNAPSHOT: Failure to find...

Seems current lstmforcaster doesn't support set different hp for different layers. It would be great if we can support that.

some examples in dllib cannot pass

1

python example status:  scala example status:  For detailed information, please refer https://github.com/intel-analytics/analytics-zoo/pull/4501 https://github.com/intel-analytics/analytics-zoo/pull/4461 Shall we update the example readme and fix the running issue.

jekins failed to catch failure of LSTMSerialTest

2

It's werid that lstm failed to save(https://github.com/intel-analytics/BigDL/issues/5200) while we have ut to cover the serialize test(https://github.com/intel-analytics/BigDL/blob/main/scala/dllib/src/test/scala/com/intel/analytics/bigdl/dllib/keras/layers/LSTMSpec.scala#L94) and our jekins can pass. I tried to run the LSTMSerialTest on local and...

convert sparkdf to pdf within arrow

1

## Description In the previous implementation, we convert rdd of spark row to pandas dataframe directly, in this pr, we convert spark row to arrow table first, then convert arrow...

Exception happened if using orca estimator train pytorch model with xshards

11

The exception is: An error occurred while calling o59.estimatorTrain. : com.intel.analytics.bigdl.dllib.utils.InvalidOperationException at com.intel.analytics.bigdl.dllib.utils.Log4Error$.invalidOperationError(Log4Error.scala:38) at com.intel.analytics.bigdl.dllib.keras.models.InternalDistriOptimizer.train(Topology.scala:1161) at com.intel.analytics.bigdl.dllib.keras.models.InternalDistriOptimizer.train(Topology.scala:1302) at com.intel.analytics.bigdl.dllib.keras.models.InternalDistriOptimizer.train(Topology.scala:969) at com.intel.analytics.bigdl.dllib.estimator.Estimator.train(Estimator.scala:192) at com.intel.analytics.bigdl.dllib.estimator.python.PythonEstimator.estimatorTrain(PythonEstimator.scala:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)...

Exception happened if using orca estimator train tensorflow.keras model with xshards of pandas dataframe

7

The exception is: ****************************Usage Error************************ model_input number does not match data number, got model_input ['dense_input'], data [TensorMeta(dtype: int64, name: list_input_0, shape: ()), TensorMeta(dtype: int64, name: list_input_1, shape: ()), TensorMeta(dtype: int64,...

If covert a spark df to shards with empty partition, will throw exception

2

## Description Fix https://github.com/intel-analytics/BigDL/issues/5877. Add logic to check if there is empty partition of spark dataframe, if there is and if `spark_df_to_pd_sparkxshards` is called, will throw exception.

[Discussion] Operations needed to be supported in shards

1

To support better user experience to use orca shards, created this issue to discuss which operations are needed to support in orca shards. - [ ] Scaler - [x] minmaxscaler...