Horovod and TF Recommenders : "AttributeError: 'VocabWeightHandler' object has no attribute 'ref'

[BrianMiner]

Hello. Thank you for this great lib and all the learning here! I am wondering if anyone has experienced the following error when trying to distribute TF Recommenders with Horovod. Im using Tensorflow 2.11 and the latest pip installed TF Recommender. It seems that Horovod tries to distribute the state of the layers from the leader node to the workers initially and its expecting there to be a ref attribute.

Some googling seems to suggest this is a Keras issue (related: https://github.com/keras-team/tf-keras/issues/650) and I am assuming, although the info is sparse, its related to the embedding layers (created here just like the TF reco example for retrieval). I thought Id ping this group to see if there are any suggestions or anyone has run into this issue, then will close the issue.

Thanks

"AttributeError: 'VocabWeightHandler' object has no attribute 'ref'