bedrockruntime/types is missing a ContentBlockMemberCachePoint

[awartoft]

Acknowledgements

• [x] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• [x] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

Hello,

I don't think this really is a bug more that the SDK is behind the bedrockruntime but it's not possible use the ConverseApi and specify a cachePoint. And it's not possible to create on one outside of the types packages because of the isContentBlock not being exported.

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

I would expect types to contain a ContentBlockMemberCachePoint and CachePoint struct that can be used.

Current Behavior

Not possible

Reproduction Steps

None

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

	github.com/aws/aws-sdk-go v1.55.5
	github.com/aws/aws-sdk-go-v2 v1.33.0
	github.com/aws/aws-sdk-go-v2/config v1.29.1
	github.com/aws/aws-sdk-go-v2/credentials v1.17.54
	github.com/aws/aws-sdk-go-v2/feature/cloudfront/sign v1.8.5
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.24
	github.com/aws/aws-sdk-go-v2/service/bedrockruntime v1.24.0
	github.com/aws/aws-sdk-go-v2/service/cloudfront v1.44.5
	github.com/aws/aws-sdk-go-v2/service/ecr v1.38.6
	github.com/aws/aws-sdk-go-v2/service/iam v1.38.7
	github.com/aws/aws-sdk-go-v2/service/ivs v1.42.6
	github.com/aws/aws-sdk-go-v2/service/mediaconvert v1.65.3
	github.com/aws/aws-sdk-go-v2/service/personalize v1.39.11
	github.com/aws/aws-sdk-go-v2/service/personalizeevents v1.25.11
	github.com/aws/aws-sdk-go-v2/service/personalizeruntime v1.27.11
	github.com/aws/aws-sdk-go-v2/service/s3 v1.74.0
	github.com/aws/aws-sdk-go-v2/service/sns v1.33.14
	github.com/aws/aws-sdk-go-v2/service/sqs v1.37.9
	github.com/aws/aws-sdk-go-v2/service/ssm v1.56.7
	github.com/aws/aws-sdk-go-v2/service/sts v1.33.9
	github.com/aws/aws-sdk-go-v2/service/translate v1.28.11
	github.com/aws/smithy-go v1.22.2

Compiler and Version used

1.23.4

Operating System and version

MacOS

[adev-code]

Hello @macnibblet, thanks for reaching out and finding out about the bug and issue. I have confirmed that the feature has been added to the API of the service but it was never released. I have reached out to the Bedrock Runtime service team for the release and will inform once there are updates. If you have any questions, please let us know. Thanks.

For Internal Use: P196027181

[awartoft]

@adev-code Sweet, thanks for the update, any eta on it being available?

[awartoft]

@adev-code it's been almost two months and no progress on this, anything we can do to help?

[KevinZhou1]

I created a similar issue here: https://github.com/aws/aws-sdk-go-v2/issues/3060

When can we expect an update from the bedrock team? This is unacceptable for an sdk to not support the needed features to get it to functionally work. Prompt caching is crucial for chat applications to reduce expensive token cost.

I also spent a lot of time getting my chat implementation to work with Converse since there weren't a lot of examples to go off of... I feel its the best language to work with for lambdas so its my go-to.

[Madrigal]

This was added as part of the v1.28.0 Bedrock release, on 03/31

You can now use cachePoint on converse as follows

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/bedrockruntime"
	"github.com/aws/aws-sdk-go-v2/service/bedrockruntime/types"
)

func main() {
	modelId := "amazon.nova-lite-v1:0" // Change to your preferred model
	prompt := "What is the capital of Mexico"

	// Load AWS configuration
	ctx := context.Background()
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion("us-east-1"),
		// uncomment this to see the raw request and response
		// config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody),
	)
	if err != nil {
		log.Fatalf("Failed to load AWS config: %v", err)
	}

	// Create Bedrock Runtime client
	client := bedrockruntime.NewFromConfig(cfg)

	// Prepare the conversation request
	input := &bedrockruntime.ConverseInput{
		ModelId: aws.String(modelId),
		Messages: []types.Message{
			{
				Role: types.ConversationRoleUser,
				Content: []types.ContentBlock{
					&types.ContentBlockMemberText{
						Value: prompt,
					},
					&types.ContentBlockMemberCachePoint{
						Value: types.CachePointBlock{
							Type: "default",
						},
					},
				},
			},
		},
		InferenceConfig: &types.InferenceConfiguration{
			MaxTokens:   aws.Int32(1000),
			Temperature: aws.Float32(0.1), // low temperature for a stable response and higher chance of cache hit
		},
	}

	// Call Bedrock
	response, err := client.Converse(ctx, input)
	if err != nil {
		log.Fatalf("Failed to call Bedrock: %v", err)
	}

	// Process the response
	fmt.Println("=== Bedrock 1st Response ===")
	fmt.Printf("Cache usage - CacheReadInputTokens: %d, CacheWriteInputTokens: %d\n",
		*response.Usage.CacheReadInputTokens, *response.Usage.CacheWriteInputTokens)

	// call the same input again, it should have a cache hit
	response, err = client.Converse(ctx, input)
	if err != nil {
		log.Fatalf("Failed to call Bedrock: %v", err)
	}

	fmt.Println("=== Bedrock 2nd Response ===")
	fmt.Printf("Cache usage - CacheReadInputTokens: %d, CacheWriteInputTokens: %d\n",
		*response.Usage.CacheReadInputTokens, *response.Usage.CacheWriteInputTokens)
	if response.Usage.CacheReadInputTokens != nil && *response.Usage.CacheReadInputTokens > 0 {
		fmt.Println("At least part of the response on the 2nd request was cached!")
	}
}

This is also available for usage with ToolMember and System cache point so you can also setup your prompt caching at that level. For more information, see the prompt caching guide

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.