[Bug]: edit_image() is unable to find any media content.

[mahsanghani]

File Name

vision/getting-started/imagen3_editing.ipynb

What happened?

built-in function edit_image() is unable to find any media content. code:

edited_image = client.models.edit_image(
    model=edit_model,
    prompt=edit_prompt,
    reference_images=[raw_ref_image, mask_ref_image],
    config=EditImageConfig(
        edit_mode="EDIT_MODE_INPAINT_INSERTION",
        number_of_images=1,
        safety_filter_level="BLOCK_LOW_AND_ABOVE",
        person_generation="DONT_ALLOW",
    ),
)

Relevant log output

code:

edited_image = client.models.edit_image(
    model=edit_model,
    prompt=edit_prompt,
    reference_images=[raw_ref_image, mask_ref_image],
    config=EditImageConfig(
        edit_mode="EDIT_MODE_INPAINT_INSERTION",
        number_of_images=1,
        safety_filter_level="BLOCK_LOW_AND_ABOVE",
        person_generation="DONT_ALLOW",
    ),
)

output:

ClientError: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Image editing failed with the following error: No uri or raw bytes are provided in media content.', 'status': 'INVALID_ARGUMENT'}}

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[katiemn]

Can you please include what your reference_images are?

[tomasdev]

I got the same error using

reference_images=[
  types.RawReferenceImage(
    reference_image=types.Image(
      image_bytes=image.generated_images[0].image.image_bytes,
      mime_type=image.generated_images[0].image.mime_type
    ),
    reference_id=0
  )
],

where image is a generate_image() response.

Nothing I tried worked, eventually it complains about reference type missing, which according to docs should be set by the SDK, not users.

[tomasdev]

Even using:

    reference_images=[
        {
            "reference_image": image.generated_images[0].image, # this is of type Image already
            "reference_id": 0
        }
    ],

throws the Reference image should have image type

if types.RawReferenceImage(reference_image=image.generated_images[0].image, reference_id=1) then it's the OP No uri or raw bytes are provided in media content

[katiemn]

Follow the steps in the "Inpainting insert" section of this notebook. It'll show you how to generate an image and use that image in an editing request.

[tomasdev]

In my case, the error was the model was incorrect. Make sure you're using an editing model for edit_image, and a generation model for generate_image

Imagen 4 does not have an edit model, so use model="imagen-3.0-capability-001"

I wish .edit_image had a validation to make sure the model specified is valid, instead of trying to send it to the model just to get a deceiving error message (missing bytes, when user has in fact sent bytes, but the model does not support editing)

[pedrojrv]

@tomasdev that was it for me!