[QST] How to deploy model using Pretrain Embeddings with EmbeddingOperator in Triton

[jhnealand]

❓ Questions & Help

Details

I have trained both a Two Tower and DLRM model using PreTrained embeddings using a Loader with EmbeddingOperator following the approach from this example:

https://github.com/NVIDIA-Merlin/models/blob/main/examples/usecases/entertainment-with-pretrained-embeddings.ipynb

My models train, and I would now like to deploy them to a Triton ensemble.

Here is the Loader I used with the EmbeddingOperator to look up the pretrained embeddings.

train_loader = mm.Loader(
    train,
    batch_size=1024,
    transforms=[
        EmbeddingOperator(
            item_embedding_array,
            lookup_key="liked",
            embedding_name="liked",
        )
    ],
)

Prior to using PreTrained Embeddings, my workflow to deploy the model looked like this:

workflow =  nvt.Workflow(['item_id'] +(['liked', .... some other informants ], 
                                       >> TransformWorkflow(retrieval_item_workflow)                                     
                                       >> PredictTensorflow(retrieval_model.candidate_encoder)))

I think I need to add a step in this workflow that looks up the pre-trained embeddings for the "liked" column. I've tried inserting an EmbeddingOperator step in the workflow directly:

workflow =  nvt.Workflow(['item_id'] +(informant_columns, 
                                       >> TransformWorkflow(retrieval_item_workflow)                                   
                                       >> EmbeddingOperator( item_embedding_array,
                                                                                lookup_key="liked",
                                                                                embedding_name="liked")
                                       >> PredictTensorflow(retrieval_model.candidate_encoder)))

I've also tried putting the EmbeddingOperator as part of my NVTabular workflow (ie. retrieval_item_workflow).

What is the correct way to deply a model that has been trained with an EmbeddingOperator transform in the Loader?

[CarloNicolini]

Maybe you can check at the documentation or examples of Merlin Systems for this?