KerasEstimator breaks for length of feature_cols > 1

[adamelk]

I was running the Keras Estimator with one feature column on an LSTM model with input shape (,50,1) passed to the feature_cols parameter, and it worked fine. The problem came when I passed it two feature columns instead for an LSTM model with input shape (,50,2).

    model = Sequential()
    model.add(LSTM(neurons, input_shape=(50, 2)))
   ...

  keras_estimator = hvd.KerasEstimator(
      num_proc=2,
      store=store,
      model=model,
      optimizer=optimizer,
      loss=loss,
      verbose=2,
      feature_cols=['features1','features2'],
      label_cols=['labels'],
      batch_size=batch_size,
      epochs=epochs)

It gave me the following error:

ValueError: Feature column count 2 must equal model inputs count 1

I dug into the traceback and this is where the issue stems from:

/databricks/python/lib/python3.9/site-packages/horovod/spark/common/util.py in check_shape_compatibility(metadata, feature_columns, label_columns, input_shapes, output_shapes, label_shapes)
    159     if input_shapes is not None:
    160         if feature_count != len(input_shapes):
--> 161             raise ValueError('Feature column count {features} must equal '
    162                              'model inputs count {inputs}'
    163                              .format(features=feature_count, inputs=len(input_shapes)))

I looked into the source code and tried to reproduce and compare the "input_shapes" variable to the input shape of my actual model and noticed that the line of code that sets the "input_shapes" variable is here (https://horovod.readthedocs.io/en/stable/_modules/horovod/spark/keras/estimator.html#KerasEstimator) :

input_shapes = [[dim if dim else -1 for dim in input.shape.as_list()]
                           for input in model.inputs]

When I ran this specific block on my model it returned "[[-1, 50, 2]]". Specifically, the len(input_shapes) will always be 1 in the code below:

if feature_count != len(input_shapes):

Maybe, the input_shape should be input_shape[0]? Please advise if there is any workaround or a different way to build the keras model with multiple input layers to match the feature_cols length.

[chongxiaoc]

@adamelk Is there a simple reproducer I can try?

[chongxiaoc]

I think the reason behind the scene is that KerasEstimator dataloader would generate same number of feature tensors and target tensors to match model input and output layers. So if input layer takes [A, B], the it expects A,B are two feature columns of Estimator.

You have to:

• Create two input layers for 'features1' and 'features2' separately.
• Or contact features1 and features2 together to be a vector input.