bug: X-Amzn-SageMaker-Custom-Attributes header conflict prevents custom value for Sagemaker endpoint

[shebna12]

Describe the bug

I would like to use the X-Amzn-SageMaker-Custom-Attributes header to get a value needed for my sagemaker endpoint. Unfortunately, according to the section #7 of the readme in https://github.com/bentoml/aws-sagemaker-deploy, it is being used by BentoML for more than two services:

Note: You can also invoke the Sagemaker endpoint directly. If there is only one service, SageMaker deployment will choose that one. If there is more than one, you can specify which service to use by passing the X-Amzn-SageMaker-Custom-Attributes header with the name of the service as value.

I receive the following error message from the CloudWatch logs:

bentoml.exceptions.BentoMLException: API Service passed via the 'X-Amzn-SageMaker-Custom-Attributes' header '' not found in the bentoml service.

Is there any way to disable the X-Amzn-SageMaker-Custom-Attributes header explicitly?

To reproduce

To reproduce a bug:

1. Deploy in sagemaker any bentoml service.
2. Send a post request with a header value for X-Amzn-SageMaker-Custom-Attributes . For example:

response = requests.post(
          sagemaker_endpoint,
          auth=AWS4Auth(
              access_id,
              access_key,
              region,
              "sagemaker",
              session_token,
          ),
          json=some_json_request,
          headers={
              "X-Amzn-SageMaker-Custom-Attributes": 'custom header value'
          },
      )

Expected behavior

I expect to pass my own custom header value to through the X-Amzn-SageMaker-Custom-Attributes so it can be used inside my bentoml service in Sagemaker Endpoint.

Environment

bentoml: 1.1.7 python: 3.9 requests: "^2.31.0" platform: AWS Sagemaker Endpoint