bedrock_agent_runtime APIs are not returning source metadata

[adoyon23]

Describe the bug

When I call the invoke_agent, retrieve, or retrieve_and_generate APIs for the bedrock_agent_runtime, the response never contains the metadata keys associated with the source files. I know that the metadata is set up correctly for the source, because I see it show up in the vector DB indexes, and if I test the knowledge base directly in the console and view the trace, it shows the correct source metadata.

However, when I call any of the APIs mentioned above referencing the same knowledge base, the response never contains the source metadata information.

Expected Behavior

The retrievedReferences key in the response from an invoke_agent call to the bedrock_agent_runtime should contain a metadata key whose values match the JSON key values set in the corresponding source .metadata.json file.

Current Behavior

There is no metadata key returned by any of the invoke_agent, retrieve, or retrieve_and_generate APIs.

Reproduction Steps

Call the invoke_agent API similar to the code below with redacted info:

def get_agent_response(response):
    if "completion" not in response:
        return f"No completion found in response: {response}"
    for event in response["completion"]:
        # Extract the traces
        if "chunk" in event:
            # Extract the bytes from the chunk
            chunk_bytes = event["chunk"]["bytes"]

            # Convert bytes to string, assuming UTF-8 encoding
            chunk_text = chunk_bytes.decode("utf-8")

            # Print the response text
            print("Response from the agent:", chunk_text)
            # If there are citations with more detailed responses, print them
            reference_text = ""
            source_file_list = []
            if (
                "attribution" in event["chunk"]
                and "citations" in event["chunk"]["attribution"]
            ):
                for citation in event["chunk"]["attribution"]["citations"]:
                    if (
                        "generatedResponsePart" in citation
                        and "textResponsePart" in citation["generatedResponsePart"]
                    ):
                        text_part = citation["generatedResponsePart"][
                            "textResponsePart"
                        ]["text"]
                        print("Detailed response part:", text_part)
                    source_file_list = []
                    metadata_values = []
                    if "retrievedReferences" in citation:
                        for reference in citation["retrievedReferences"]:
                            print("Reference")
                            print(reference)
                            if (
                                "content" in reference
                                and "text" in reference["content"]
                            ):
                                reference_text = reference["content"]["text"]
                                print("Reference text:", reference_text)
                            if "location" in reference:
                                source_file = reference["location"]["s3Location"]["uri"]
                                source_file_list.append(source_file)
                            if "metadata" in reference:
                                print("Found metadata")
                                print(reference["metadata"])
                                print(reference["metadata"]["string"])
                                metadata_values = reference["metadata"]["string"]
                    print(f"source_file_list: {source_file_list}")
                    print(f"meta_data_values_list: {metadata_values}")

    client = boto3.client("bedrock-agent-runtime", region_name="us-west-2")

    response = client.invoke_agent(
         agentAliasId='***',
         agentId='***',
         sessionId="session1",
         inputText="inputText",
        enableTrace=True
    )

  chunk_text, reference_text, source_file_list = get_agent_response(response)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3-1.34.99

Environment details (OS name and version, etc.)

Mac OS 14.4.1 (23E224), Python 3.12 runtime for AWS Lambda