TypeError: compile() got an unexpected keyword argument 'optimizer' , AttributeError: 'TorchModuleWrapper' object has no attribute 'fit'

[openSourcerer9000]

As I understand it, TorchModuleWrapper ports an nn.Module to a Keras model, no? Trying to fine-tune the grounding dino model, I first set up the nn.Module (an inherited subclass anyway), and then wrap it with TorchModuleWrapper. Then copying code, directly from the only documentation I can find on using a pytorch backend, I come to find out it throws an error trying to use .compile() or .fit() methods. I'm now thoroughly confused. Does it not actually work with pytorch, or is there some more complete documentation on how to port a pytorch model over to keras 3? Does everything needs to be pytorch specific for TorchModuleWrapper? If so, then what benefit does keras provide when the model is in pytorch?

keras v '3.0.4'

import os
os.environ["KERAS_BACKEND"] = "torch"

import groundingdino.datasets.transforms as T
from groundingdino.models import build_model
from groundingdino.util import box_ops
from groundingdino.util.inference import predict
from groundingdino.util.slconfig import SLConfig
from groundingdino.util.utils import clean_state_dict
from huggingface_hub import hf_hub_download

import torch
import torch.nn as nn
import torch.nn.functional as F

import keras
from keras.layers import TorchModuleWrapper
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# 'cuda'

def load_model_hf(
    repo_id: str, filename: str, ckpt_config_filename: str, device: str = "cpu"
) -> torch.nn.Module:
    """
    Loads a model from HuggingFace Model Hub.

    Args:
        repo_id (str): Repository ID on HuggingFace Model Hub.
        filename (str): Name of the model file in the repository.
        ckpt_config_filename (str): Name of the config file for the model in the repository.
        device (str): Device to load the model onto. Default is 'cpu'.

    Returns:
        torch.nn.Module: The loaded model.
    """

    cache_config_file = hf_hub_download(
        repo_id=repo_id,
        filename=ckpt_config_filename,
        force_filename=ckpt_config_filename,
    )
    args = SLConfig.fromfile(cache_config_file)
    model = build_model(args)
    model.to(device)
    cache_file = hf_hub_download(
        repo_id=repo_id, filename=filename, force_filename=filename
    )
    checkpoint = torch.load(cache_file, map_location="cpu")
    model.load_state_dict(clean_state_dict(checkpoint["model"]), strict=False)
    model.eval()
    return model
def build_groundingdino():
    """Build the GroundingDINO model."""
    ckpt_repo_id = "ShilongLiu/GroundingDINO"
    ckpt_filename = "groundingdino_swinb_cogcoor.pth"
    ckpt_config_filename = "GroundingDINO_SwinB.cfg.py"
    groundingdino = load_model_hf(
        ckpt_repo_id, ckpt_filename, ckpt_config_filename, device
    )
    return groundingdino
dino = build_groundingdino()
assert isinstance(dino,nn.Module)
model = TorchModuleWrapper(dino)
type(model)

model.compile(
    loss="sparse_categorical_crossentropy",
    optimizer="adam",
    metrics=["accuracy"]
)
# Throws: TypeError: compile() got an unexpected keyword argument 'loss'

model.compile(
    optimizer='adam', 
    loss='sparse_categorical_crossentropy', 
    metrics=['accuracy']  
)
# Throws: TypeError: compile() got an unexpected keyword argument 'optimizer'

model.fit
#Throws: AttributeError: 'TorchModuleWrapper' object has no attribute 'fit'

[fchollet]

The issue here is that you are attempting to model = TorchModuleWrapper(dino) as if it were a Keras Model. In reality, it is a Layer, so it doesn't have the fit() method (and the compile() method you're hitting is the torch one).

You should instead create a Model subclass that calls dino inside call(). Here's a simple example of using a PyTorch module inside a Keras Model: https://colab.research.google.com/drive/1tWJNm3g_EnRdZIkKZk_FGwUU7LMf8vb5#scrollTo=aCZO64R5Ol-k

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[openSourcerer9000]

Gotcha, thanks. So would TorchModuleWrapper work with the functional or sequential API?

I'm not at a computer now but something like this?

img_input = keras.Input(shape=(256,256 3))
text_input = keras.Input(shape=(1,), dtype=tf.string, name='text')

Dinolyr = TorchModuleWrapper(dino)

x = dinolyr(img_input,text_input)
Output = myHead(x) # sequential model of custom top layers 

model = Model( (img_input,text_input) , output)