promptfoo icon indicating copy to clipboard operation
promptfoo copied to clipboard

Unable to pass images to Llama 3.2 on Bedrock

Open decaffeinatedio opened this issue 8 months ago • 3 comments

Describe the bug When I implement the Nova template (https://github.com/promptfoo/promptfoo/blob/main/examples/amazon-bedrock/nova_multimodal_prompt.json) for Llama 3.2 on bedrock, I encounter the error:

[ERROR] TypeError: message.content.trim is not a function

To Reproduce Use https://github.com/promptfoo/promptfoo/blob/main/examples/amazon-bedrock/nova_multimodal_prompt.json, but configure the provider to point to a bedrock:<LLAMA ARN>. Pass and reference an {{image}} var.

Expected behavior Base64 image is passed to Llama on bedrock for evaluation

Screenshots

System information:

====================================================================================================
Promptfoo Debug Information
====================================================================================================
{
  "version": "0.109.1",
  "platform": {
    "os": "linux",
    "release": "6.11.0-21-generic",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
====================================================================================================

- Promptfoo version: 0.109.1

**Additional context**

decaffeinatedio avatar Apr 11 '25 15:04 decaffeinatedio

@decaffeinatedio have you tried running the example located in examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml?

When I run that example which uses the prompt you reference, I get Image

mldangelo avatar Apr 11 '25 15:04 mldangelo

@mldangelo Yep! I have no problems running the example against Nova. But when I use the bedrock Llama provider:

  - id: bedrock:arn:aws:bedrock:us-east-1:<<REDACTED>>:inference-profile/us.meta.llama3-2-11b-instruct-v1:0
    config: 
      max_gen_len: 256
      temperature: 0.7
      top_p: 0.9

I'm encountering that error. When I simplify my prompt to text-only, I get responses back from Llama, so I know it isn't a broader connection/permissions issue.

decaffeinatedio avatar Apr 11 '25 16:04 decaffeinatedio

@mldangelo took a quick look at source and this doesn't seem straightforward to address. Assume it won't be addressed in the short term?

decaffeinatedio avatar Apr 30 '25 23:04 decaffeinatedio