Botocore FIPS Support

[hazedav]

Botocore FIPS Analysis

References

https://aws.amazon.com/compliance/fips/ https://docs.aws.amazon.com/govcloud-us/latest/UserGuide/using-govcloud-endpoints.html

Use Case

Given a "standard" AWS region name (i.e. us-east-1) and service name (i.e. kms), determine the FIPS region name which can be passed to the client for proper endpoint URI resolution (i.e. fips-us-east-1).

Problem Statement

I have discovered a number of challenges with get_available_regions() which makes selecting a corresponding FIPS region untenable (see Apendix A). My understanding is that get_available_regions() is driven by a programatically generated endpoint.json file.

There are 4 distinct flavors of issues that I was able to uncover:

• Cases where none of the non_regional fips regions are represented in endpoints.json
• Cases where the service is missing from get_available_services()
• Cases where a single region is represented as an ambiguous non_regional "fips" and the other fips regions are missing.

For instance appstream2 should have 4 specific non_regional regions but endpoints.json simply has:

"fips" : {
  "credentialScope" : {
    "region" : "us-west-2"
  },
  "hostname" : "appstream2-fips.us-west-2.amazonaws.com"
},

The fips non_regional naming is problematic for any service that has multiple regions because it's ambigous as to what region this fips equivelant endpoint applies to.

• Cases where using-govcloud-endpoints lists the same URIs as non-FIPS and compliance/fips lists them as FIPS :O

For instance:

appconfig.us-gov-west-1.amazonaws.com
appconfig.us-gov-east-1.amazonaws.com

Appendix A

Service	Partition	Endpoints	Reason
apigateway	aws	apigateway-fips.us-east-1.amazonaws.com apigateway-fips.us-east-2.amazonaws.com apigateway-fips.us-west-1.amazonaws.com apigateway-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
appstream	aws	appstream2-fips.us-east-1.amazonaws.com appstream2-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
clouddirectory	aws	clouddirectory-fips.us-east-1.amazonaws.com clouddirectory-fips.us-east-2.amazonaws.com clouddirectory-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
cloudfront	aws	cloudfront-fips.amazonaws.com	Not represented in endpoints.json
cognito-sync	aws	cognito-sync-fips.us-east-1.amazonaws.com cognito-sync-fips.us-east-2.amazonaws.com cognito-sync-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
connect	aws	connect-fips.us-east-1.amazonaws.com connect-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
connectparticipant	aws	participant.connect.us-east-1.amazonaws.com participant.connect-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
imagebuilder	aws	imagebuilder-fips.us-east-1.amazonaws.com imagebuilder-fips.us-east-2.amazonaws.com imagebuilder-fips.us-west-1.amazonaws.com imagebuilder-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
elasticache	aws	elasticache-fips.us-east-1.amazonaws.com elasticache-fips.us-east-2.amazonaws.com elasticache-fips.us-west-1.amazonaws.com elasticache-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
es	aws	es-fips.us-east-1.amazonaws.com es-fips.us-east-2.amazonaws.com es-fips.us-west-1.amazonaws.com es-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
inesisanalytics	aws	kinesisanalytics-fips.us-east-1.amazonaws.com kinesisanalytics-fips.us-east-2.amazonaws.com kinesisanalytics-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
lexv2-runtime	aws	runtime-v2-lex-fips.us-east-1.amazonaws.com runtime-v2-lex-fips.us-east-1.amazonaws.com	Not represented in endpoints.json
kafka	aws	kafka-fips.us-west-1.amazonaws.com kafka-fips.us-west-2.amazonaws.com kafka-fips.us-east-2.amazonaws.com kafka-fips.us-east-1.amazonaws.com	Not represented in endpoints.json
quicksight	aws	fips-us-east-1.quicksight.aws.amazon.com fips-us-east-2.quicksight.aws.amazon.com fips-us-west-2.quicksight.aws.amazon.com	Not represented in endpoints.json
rds-data	aws	rds-data-fips.us-east-1.amazonaws.com rds-data-fips.us-east-2.amazonaws.com rds-data-fips.us-west-1.amazonaws.com rds-data-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
ses	aws	email-fips.us-east-1.amazonaws.com email-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
s3	aws	s3-fips.us-east-1.amazonaws.com s3-fips.us-east-2.amazonaws.com s3-fips.us-west-1.amazonaws.com s3-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
textract	aws	textract-fips.us-east-1.amazonaws.com textract-fips.us-east-2.amazonaws.com textract-fips.us-west-1.amazonaws.com textract-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
timestream-query	aws	query.timestream-fips.us-east-1.amazonaws.com query.timestream-fips.us-east-2.amazonaws.com query.timestream-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
timestream-write	aws	ingest.timestream-fips.us-east-1.amazonaws.com ingest.timestream-fips.us-east-2.amazonaws.com ingest.timestream-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
transcribestreaming	aws	transcribestreaming-fips.us-east-1.amazonaws.com transcribestreaming-fips.us-east-2.amazonaws.com transcribestreaming-fips.us-west-2.amazonaws.com	This service isn't recognized in get_available_services()
backup	aws	backup-fips.us-east-1.amazonaws.com backup-fips.us-east-2.amazonaws.com backup-fips.us-west-1.amazonaws.com backup-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
servicediscovery	aws	servicediscovery-fips.us-east-1.amazonaws.com servicediscovery-fips.us-east-2.amazonaws.com servicediscovery-fips.us-west-1.amazonaws.com servicediscovery-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "servicediscovery-fips"
codecommit	aws	codecommit-fips.us-east-1.amazonaws.com codecommit-fips.us-east-2.amazonaws.com codecommit-fips.us-west-1.amazonaws.com codecommit-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
dms	aws	dms-fips.us-east-1.amazonaws.com dms-fips.us-east-2.amazonaws.com dms-fips.us-west-1.amazonaws.com dms-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "dms-fips"
iotsecuretunneling	aws	api.tunneling.iot-fips.us-east-1.amazonaws.com api.tunneling.iot-fips.us-east-2.amazonaws.com api.tunneling.iot-fips.us-west-1.amazonaws.com api.tunneling.iot-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
kms	aws	kms-fips.us-east-1.amazonaws.com kms-fips.us-east-2.amazonaws.com kms-fips.us-west-1.amazonaws.com kms-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
opsworkscm	aws	opsworks-cm-fips.us-east-1.amazonaws.com opsworks-cm-fips.us-east-2.amazonaws.com opsworks-cm-fips.us-west-1.amazonaws.com opsworks-cm-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
storagegateway	aws	storagegateway-fips.us-east-1.amazonaws.com storagegateway-fips.us-east-2.amazonaws.com storagegateway-fips.us-west-1.amazonaws.com storagegateway-fips.us-west-2.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
wafv2	aws	wafv2-fips.us-east-2.amazonaws.com wafv2-fips.us-west-1.amazonaws.com wafv2-fips.us-east-1.amazonaws.com wafv2-fips.us-west-2.amazonaws.com	Not represented in endpoints.json
appconfig	aws-us-gov	appconfig.us-gov-east-1.amazonaws.com appconfig.us-gov-west-1.amazonaws.com	Conflicting FIPS specification (both FIPS and non-FIPS)
apigateway	aws-us-gov	apigateway-fips.us-gov-east-1.amazonaws.com apigateway-fips.us-gov-west-1.amazonaws.com	Not represented in endpoints.json
backup	aws-us-gov	backup-fips.us-gov-east-1.amazonaws.com backup-fips.us-gov-west-1.amazonaws.com	Not represented in endpoints.json
cloudhsmv2	aws-us-gov	?	Not represented in endpoints.json
cloudhsm	aws-us-gov	?	Not represented in endpoints.json
servicediscovery	aws-us-gov	servicediscovery-fips.us-gov-east-1.amazonaws.com servicediscovery-fips.us-gov-west-1.amazonaws.com	Misrepresented as ambiguous non_regional "servicediscovery-fips"
codecommit	aws-us-gov	codecommit-fips.us-gov-west-1.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
opensearch	aws-us-gov	es-fips.us-gov-east-1.amazonaws.com es-fips.us-gov-west-1.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
databrew	aws-us-gov	?	Not represented in endpoints.json
greengrass	aws-us-gov	greengrass.us-gov-east-1.amazonaws.com greengrass.us-gov-west-1.amazonaws.com	fips-us-gov-west-1 not represented in endpoints.json
greengrass2	aws-us-gov	greengrass.us-gov-east-1.amazonaws.com greengrass.us-gov-west-1.amazonaws.com	fips-us-gov-west-1 not represented in endpoints.json
iot-defender	aws-us-gov	?	This service is listed as having non-FIPS endpoints but is not recognized in get_available_services()
kendra	aws-us-gov	?	Not represented in endpoints.json
kms	aws-us-gov	kms-fips.us-gov-east-1.amazonaws.com kms-fips.us-gov-west-1.amazonaws.com	Misrepresented as ambiguous non_regional "ProdFips"
kafka	aws-us-gov	kafka.us-gov-east-1.amazonaws.com kafka.us-gov-west-1.amazonaws.com	Conflicting FIPS specification (both FIPS and non-FIPS)
phd	aws-us-gov	?	This service is listed as having non-FIPS endpoints but is not recognized in get_available_services()
ram	aws-us-gov	ram.us-gov-west-1.amazonaws.com ram.us-gov-east-1.amazonaws.com	Conflicting FIPS specification (both FIPS and non-FIPS)
s3	aws-us-gov	s3-fips.us-gov-east-1.amazonaws.com s3-fips.us-gov-west-1.amazonaws.com	fips-us-gov-east-1 not represented in endpoints.json
storagegateway	aws-us-gov	storagegateway-fips.us-gov-east-1.amazonaws.com storagegateway-fips.us-gov-west-1.amazonaws.com	Misrepresented as ambiguous non_regional "fips"
textract	aws-us-gov	textract-fips.us-gov-east-1.amazonaws.com textract-fips.us-gov-west-1.amazonaws.com	Not represented in endpoints.json
wafv2	aws-us-gov	wafv2-fips.us-gov-east-1.amazonaws.com wafv2-fips.us-gov-west-1.amazonaws.com	Not represented in endpoints.json

[stobrien89]

Hi @hazedav,

Thank you for pointing this out! We appreciate the detailed analysis.

For the services not listed when using get_available_services, the models for those services don't exist in botocore at this time. Service models are generated upstream and are pushed to the SDKs when SDK support is ready/available. Updates to service models are pushed to the SDKs as well.

Similarly, the endpoints available for a given service are provided and updated by service teams. As you pointed out, there are a number of services that have not provided updated FIPS endpoints. In the past we've handled this on a per-case basis, meaning we've addressed missing information with each service team individually by submitting internal tickets. In this case, I don't think it would be a good use of time to submit ±50 individual internal tickets, so I'm going to see if we can tackle this in more of a sweeping manner.

[rmharwood]

Any update on this at all? I'm working in US GovCloud and with fips enabled and coming across problems that are related to this issue. I have had to adjust the IAM section of endpoints.json as follows, to add a credentialScope to the fips variant:

      "iam" : {
        "endpoints" : {
          "aws-us-gov-global" : {
            "credentialScope" : {
              "region" : "us-gov-west-1"
            },
            "hostname" : "iam.us-gov.amazonaws.com",
            "variants" : [ {
              "credentialScope" : {
                "region" : "us-gov-west-1"
              },
              "hostname" : "iam.us-gov.amazonaws.com",
              "tags" : [ "fips" ]
            } ]
          }
        },
        "isRegionalized" : false,
        "partitionEndpoint" : "aws-us-gov-global"
      },

(I took out the deprecated statements for brevity. ) In addition, I had to add a fips variant to the EC2 section. The IAM and EC2 service endpoints default to fips, there is no alternative. There are probably other services like this, that I have not checked into. I'm not sure my workarounds are the most sensible, I just needed it to work.

[seanorama]

ec2-instance-connect is also broken in govcloud:

$ AWS_USE_FIPS_ENDPOINT=true aws ec2-instance-connect ...
Could not connect to the endpoint URL: "https://ec2-instance-connect-fips.us-gov-west-1.amazonaws.com/"

1. In GovCloud, the default endpoint is already FIPS (e.g., ec2-instance-connect.us-gov-west-1.amazonaws.com)
2. So I assume they decided not to make ec2-instance-connect-fips
3. But that then requires all SDKs and users to change logic.

For now, the SDKs have to adapt to it. But AWS would reduce waste by following convention:

1. Always have FIPS endpoints available at -fips even if the default already has FIPS (throw a CNAME or duplicate the DNS record).

[jonemo]

The following is a response to the original issue:

Given a "standard" AWS region name (i.e. us-east-1) and service name (i.e. kms), determine the FIPS region name which can be passed to the client for proper endpoint URI resolution (i.e. fips-us-east-1).

The recommended way to connect to a FIPS-compliant endpoint in a given region is to use the use_fips_endpoint configuration setting. For example, to connect to the FIPS endpoint for Amazon Athena in us-west-2:

import botocore.session
from botocore.config import Config

session = botocore.session.get_session()

client_with_fips = session.create_client(
   "athena",
    config=Config(region_name="us-west-2", use_fips_endpoint=True),
)

Alternatively, the AWS_USE_FIPS_ENDPOINT environment variable can be used to enable this setting. The comment above shows an example for this.

---

While the method outlined in the issue description is possible, it has several drawbacks:

1. The use of "FIPS pseudo regions", for example fips-us-east-1, is deprecated. References to FIPS pseudo regions should no longer exist in the documentation and all pseudo region names are labeled as deprecated in endpoints.json. Please let us know if you find an occurrence where that is not the case.
2. Starting with version 1.28.0, botocore uses a new implementation of endpoint resolution that no longer relies on endpoints.json. Each service now has an endpoint-rule-set-1.json file in its botocore/data subdirectory. This ruleset file defines the logic for the service's endpoint resolution, including operation-specific and other complex endpoint resolution logic that botocore previously applied on top of the result obtained from endpoints.json. Today, the endpoints.json file and get_available_regions() remain in botocore only for backwards compatibility and informational purposes.
3. Manually specifying an endpoint with endpoint_url is only recommended when working with account-specific or otherwise non-standard endpoints. The most common example are VPC endpoints. However, setting an endpoint_url overrides most operation-specific endpoint resolution logic and causes other config settings that would normally affect endpoint resolution to be ignored.

[jonemo]

Before adding comments to this issue, please consider these two related topics:

• We are currently investigating why there are cases where botocore's FIPS endpoint data is inconsistent with the documentation and, in most of those cases, incorrect. While we are getting to the bottom of this, please use https://github.com/boto/botocore/issues/2833 to report any additional examples.
• Decisions about availability and name of endpoints are made independently by each AWS service. For feedback and suggestions about services' endpoints, we recommend reaching out through AWS Support if you have an AWS Support plan.