BertPreprocessor raises AttributeError with keras.Input

[calvingiles]

Describe the bug When trying to create a keras model using an input with BertPreprocessor, it is impossible to construct. This is based off of following Sentence embeddings using Siamese RoBERTa-networks and switching to Bert.

To Reproduce

import keras_nlp
from tensorflow import keras

inputs = keras.Input(shape=(1), dtype="string", name="sentence")

preprocessor = keras_nlp.models.BertPreprocessor.from_preset("bert_tiny_en_uncased")

preprocessor(inputs)

results in:

AttributeError                            Traceback (most recent call last)
Cell In[78], line 8
      4 inputs = keras.Input(shape=(1), dtype="string", name="sentence")
      6 preprocessor = keras_nlp.models.BertPreprocessor.from_preset("bert_tiny_en_uncased")
----> 8 preprocessor(inputs)

...


    
        AttributeError: 'function' object has no attribute 'shape'
    
    
    Call arguments received by layer 'multi_segment_packer_13' (type MultiSegmentPacker):
      • inputs=['tf.RaggedTensor(values=tf.RaggedTensor(values=Tensor("bert_preprocessor_6/bert_tokenizer_7/FastWordpieceTokenizeWithOffsets/FastWordpieceTokenizeWithOffsets/Cast:0", shape=(None,), dtype=int32, device=/device:CPU:*), row_splits=Tensor("bert_preprocessor_6/bert_tokenizer_7/RaggedFromRowSplits_3/control_dependency:0", shape=(None,), dtype=int64, device=/device:CPU:*)), row_splits=Tensor("bert_preprocessor_6/bert_tokenizer_7/RaggedFromUniformRowLength/control_dependency:0", shape=(None,), dtype=int64, device=/device:CPU:*))']


Call arguments received by layer "bert_preprocessor_6" (type BertPreprocessor):
  • x=tf.Tensor(shape=(None, 1), dtype=string)
  • y=None
  • sample_weight=None

Using:

python==3.10
keras==2.13.1
keras-core==0.1.5
keras-nlp==0.6.1

Expected behavior It should be possible to call x = preprocessor(inputs) and then use x as part of defining a model.

Additional context

I have confirmed this issue on python 3.10 and 3.11, and with Bert and DistilBert (which use WordPiece), but not Roberta (Which uses BytePair). When swaping tokenizers and packers, it seems the issues is likely related to the BertTokenizer returning an invalid type. BertTokenizer returns:

<KerasTensor: type_spec=RaggedTensorSpec(TensorShape([None, None, None]), tf.int32, 2, tf.int64) (created by layer 'bert_tokenizer_4')>

while RobertaTokenizer returns:

<KerasTensor: type_spec=RaggedTensorSpec(TensorShape([None, None]), tf.int32, 1, tf.int64) (created by layer 'roberta_tokenizer_3')>

Would you like to help us fix it? Sure.

Full attribute error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[57], line 1
----> 1 roberter_preprocessor.packer(bert_preprocessor.tokenizer(inputs))

File /opt/conda/envs/py310/lib/python3.10/site-packages/keras_nlp/src/layers/preprocessing/preprocessing_layer.py:37, in PreprocessingLayer.__call__(self, *args, **kwargs)
     33 def __call__(self, *args, **kwargs):
     34     # Always place on CPU for preprocessing, to avoid expensive back and
     35     # forth copies to GPU before the trainable model.
     36     with tf.device("cpu"):
---> 37         outputs = super().__call__(*args, **kwargs)
     39         # Jax and Torch lack native string and ragged types.
     40         # If we are running on those backends and not running with tf.data
     41         # (we are outside a tf.function), we covert all ragged and string
     42         # tensor to pythonic types.
     43         is_tf_backend = config.backend() == "tensorflow"

File /opt/conda/envs/py310/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File /tmp/__autograph_generated_fileaavrvsc1.py:10, in outer_factory.<locals>.inner_factory.<locals>.tf__call(self, inputs)
      8 do_return = False
      9 retval_ = ag__.UndefinedReturnValue()
---> 10 (inputs, unbatched) = ag__.converted_call(ag__.ld(self)._sanitize_inputs, (ag__.ld(inputs),), None, fscope)
     11 segments = ag__.converted_call(ag__.ld(self)._trim_inputs, (ag__.ld(inputs),), None, fscope)
     12 (token_ids, segment_ids) = ag__.converted_call(ag__.ld(self)._combine_inputs, (ag__.ld(segments),), None, fscope)

File /tmp/__autograph_generated_fileqrymo7te.py:40, in outer_factory.<locals>.inner_factory.<locals>.tf___sanitize_inputs(self, inputs)
     38     pass
     39 ag__.if_stmt(ag__.not_(ag__.ld(inputs)), if_body_1, else_body_1, get_state_1, set_state_1, (), 0)
---> 40 (inputs, unbatched_list, _) = ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(zip), tuple((ag__.converted_call(ag__.ld(convert_to_ragged_batch), (ag__.ld(x),), None, fscope) for x in ag__.ld(inputs))), None, fscope),), None, fscope)
     42 def get_state_2():
     43     return ()

File /tmp/__autograph_generated_fileqrymo7te.py:40, in <genexpr>(.0)
     38     pass
     39 ag__.if_stmt(ag__.not_(ag__.ld(inputs)), if_body_1, else_body_1, get_state_1, set_state_1, (), 0)
---> 40 (inputs, unbatched_list, _) = ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(zip), tuple((ag__.converted_call(ag__.ld(convert_to_ragged_batch), (ag__.ld(x),), None, fscope) for x in ag__.ld(inputs))), None, fscope),), None, fscope)
     42 def get_state_2():
     43     return ()

File /tmp/__autograph_generated_filea97ota7r.py:124, in outer_factory.<locals>.inner_factory.<locals>.tf__convert_to_ragged_batch(inputs)
    122 def else_body_5():
    123     pass
--> 124 ag__.if_stmt(ag__.or_(lambda : ag__.ld(inputs).shape.rank < 1, lambda : ag__.ld(inputs).shape.rank > 2), if_body_5, else_body_5, get_state_5, set_state_5, (), 0)
    125 unbatched = ag__.ld(inputs).shape.rank == 1
    126 rectangular = ag__.or_(lambda : ag__.ld(rectangular), lambda : ag__.ld(unbatched))

File /tmp/__autograph_generated_filea97ota7r.py:120, in outer_factory.<locals>.inner_factory.<locals>.tf__convert_to_ragged_batch.<locals>.if_body_5()
    119 def if_body_5():
--> 120     raise ag__.converted_call(ag__.ld(ValueError), (f'Tokenized tensor input should be rank 1 (unbatched) or rank 2 (batched). Received: `inputs.shape={ag__.ld(input).shape}`',), None, fscope)

AttributeError: Exception encountered when calling layer "multi_segment_packer_9" (type MultiSegmentPacker).

in user code:

    File "/opt/conda/envs/py310/lib/python3.10/site-packages/keras_nlp/src/layers/preprocessing/multi_segment_packer.py", line 281, in call  *
        inputs, unbatched = self._sanitize_inputs(inputs)
    File "/opt/conda/envs/py310/lib/python3.10/site-packages/keras_nlp/src/layers/preprocessing/multi_segment_packer.py", line 206, in _sanitize_inputs  *
        inputs, unbatched_list, _ = list(
    File "/opt/conda/envs/py310/lib/python3.10/site-packages/keras_nlp/src/utils/tensor_utils.py", line 113, in convert_to_ragged_batch  *
        raise ValueError(

    AttributeError: 'function' object has no attribute 'shape'


Call arguments received by layer "multi_segment_packer_9" (type MultiSegmentPacker):
  • inputs=tf.RaggedTensor(values=tf.RaggedTensor(values=Tensor("Placeholder:0", shape=(None,), dtype=int32), row_splits=Tensor("Placeholder_1:0", shape=(None,), dtype=int64)), row_splits=Tensor("Placeholder_2:0", shape=(None,), dtype=int64))

[mattdangerw]

Do you have a colab repro available? That would help us dig in.

[calvingiles]

Sure - here https://colab.research.google.com/drive/1gJSfUc9YwOxBAzviSGqmLQgS2q-uehj1#scrollTo=oTsyVxGFqGVt

[calvingiles]

Looks like the correct way to do this is:

import keras_nlp
from tensorflow import keras

inputs = keras.Input(shape=(), dtype="string")
preprocessor = keras_nlp.models.BertPreprocessor.from_preset("bert_tiny_en_uncased")
preprocessor(inputs)

for single sentences. It is possible to pass a list of lists and have the proprocessor pack them, but I haven't found a way to make this work with the keras functional api yet.

[sonali-kumari1]

Hi @calvingiles -

Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved!

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.