REST API: Provide a way to add extra JSON to generated OpenAPI schema endpoints

[nlykkei]

Use case

It would be helpful, if you would support adding extra JSON to generated OpenAPI schema endpoints.

We would like to use the generated OpenAPI scheme to provision an AWS API Gateway REST API using a OpenAPI 3.0 spec. For this, we need to add e.g. x-amazon-apigateway-integration objects.

The only way to accomplish this is by mutating the generated API spec using a post-processing?

https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#customizing-api-operations

Example:

  /presigned_put_url:
    get:
      tags:
        - client support
      operationId: getPresignedPutUrl
      description: returns a url for uploading files to the bff backend
      parameters:
        - in: query
          name: file_name
          description: Name of the destination file
          example: my_file.txt
          schema:
            type: string
      responses:
        <<: *default-responses
        "200":
          description: 200 New content - returned when new content available
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/PutUrlResponse"
      x-amazon-apigateway-integration:
        <<: *apigw-integration-201
        uri: arn:aws:apigateway:${region}:lambda:path/2015-03-31/functions/${presigned_put_arn}/invocations
      security: *oidc-security

Solution/User Experience

I would like to add extra JSON at the endpoint level, using e.g. extra_json, to embed custom AWS API Gateway objects, where they are used

Alternative solutions

I could manipulate the generated OpenAPI spec myself using post-processing

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (Python) Tenets
• [ ] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[boring-cyborg[bot]]

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hello @rubenfonseca! I would like to hear your opinion here. To me, it makes some sense we inject some additional JSON/keys into the OpenAPI specification, but I'm not sure if we are breaking the OpenAPI contract and may have additional problems with this.

[rubenfonseca]

@leandrodamascena Yes, you can add custom keys prefixed with "x-" in an OpenAPI schema while still maintaining its validity. These keys, also known as "vendor extensions," allow you to include additional metadata or vendor-specific information in the schema without violating the OpenAPI specification.

The OpenAPI specification allows for the use of vendor extensions, which are properties that begin with the prefix "x-". These extensions can be added at various levels of the schema, such as the root level, object definitions, operation objects, and more. So this would break anything.

It's a nice feature that is heavily explorer by @aws/pdk for instance

[rubenfonseca]

@nlykkei I was thinking adding a new optional parameter to all the HTTP method annotations, for instance:

@app.get("/hello", vendor_extensions={...}), where vendor_extensions is of type Optional[dict[str, Any]]

Question: does this make sense to you?

I'm also considering what levels should this be added to. Documentation says:

Extensions are supported on the root level of the API spec and in the following places:

• info section
• paths section, individual paths and operations
• operation parameters
• responses
• tags
• security schemes

Following the same approach, we could add a vendor_extensions optional field to all those places.

[nlykkei]

Thank you, Ruben.

I'm also considering what levels should this be added to. Documentation says:

Which documentation are you referring to?

Yes, it would sense to add vendor_extensions to those places, so users can inject custom JSON into the generated spec, like the x-amazon-apigateway-integrationobject above.

It could also be used to inject other means of authentication, like API key, HTTP Basic, etc. OAuth 2.0 doesn't cover all use cases :)

[rubenfonseca]

Sorry, forgot the link to the documentation!

I would prefer not to abuse this mechanism to just add random things to the schema :P For security specifically, please see the plan I shared in the other thread!

If you want to take 1 (or more) of this tasks, please let me know :D otherwise I'll be happy to help

[rubenfonseca]

@nlykkei I'm planning to work on this tomorrow if that's ok :)

[nlykkei]

@rubenfonseca Thank you 🎉

Consider Pydantic's Field, it has an attribute called json_schema_extra for providing extra JSON schema properties:

A dict or callable to provide extra JSON schema properties.

https://docs.pydantic.dev/latest/api/fields/

Such customizability is important for injecting e.g. AWS API Gateway specific objects into the generated OpenAPI spec. Furthermore, depending on the securitySchemes and security implementation for OAuth 2.0, this level of customizability will also be required for users to generate other types of OpenAPI security mechanisms, e.g. API key, HTTP Basic, etc.

I would prefer not to abuse this mechanism to just add random things to the schema :P For security specifically, please see the plan I shared in https://github.com/aws-powertools/powertools-lambda-python/issues/4036!

I wouldn't think to much about abusing this mechanism, as neither does Pydantic. It's a mean for users to extend Powertools for AWS Lambda OpenAPI generation capabilities.

I'm really sorry that I haven't had more time. I'm an open source developer by heart, thus my current job takes too much of my time

[nlykkei]

Sorry, forgot the link to the documentation!

It shows an example of embedding API Gateway authorizer extension to have API Gateway create a custom Lambda authorizer using the OpenAPI spec

[rubenfonseca]

I think your point of using json_schema_extra is EXCELLENT for this feature! Thank you so much for pointing this out!

[nlykkei]

I'm happy you found it useful! :) I think Pydantic serves as a good reference

[EricOddz]

Hello, is there any news about this issue?

[leandrodamascena]

Hi @EricOddz! Not yet, I'm working on some v3 issues, but I'll try to look into it next Monday, ok?

[EricOddz]

thank you so much!!🚀

[leandrodamascena]

Hey, I need help here to confirm whether we are going on the right path or not. I read this thread again a few times and it seems that we are assuming that vendor/openapi extensions are only supported in tags, paths (all levels), operations, parameters, responses and security, as this is what is described in the Swagger example.

But I'm looking at the OpenAPI 3.0 and OpenAPI 3.1 schemas definition and it seems that vendor extensions are accepted at practically all levels of the OpenAPI schema, including root. This brings me some concerns:

1 - Currently we are unable to differentiate the Root and Info object, that is, if the customer creates a definition in get_openapi_json_schema function it will assume the same values for Info and Root, I find this confusing, don't you? I think we should assume to add this information in the info section and forget about Root.

app.py

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.utilities.typing import LambdaContext

app = APIGatewayRestResolver(enable_validation=True)


@app.get("/hello")
def get_todo_title() -> str:
    return app.get_openapi_json_schema(vendor_extensions={"x-project":  {"powertools": "x"}})


def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

2 - We cannot define an extension in the Root of "paths", but I believe it is possible to define it per operation and per operation parameter.

I propose that we adopt vendor extensions in the following sections and inform in our documentation that only these are supported. From the analysis I did, this covers most of the use cases and some tools that create some type of vendor extension:

• info
• path operation
• path parameters
• response
• security schemes

I would like your opinion here.

Thanks

[dywilson-firstam]

I'd like to chime in with some additional examples we use these for specifically for api gateway:

• servers
• root attributes

[leandrodamascena]

Thanks a lot for your contribution @dywilson-firstam, looks like we have more cases to cover. I'm adding the tag researching and I'm going to do some exploratory testing of all the scenarios we need to cover to get this working in a way that helps our customers.

[leandrodamascena]

We plan to include this support in the release scheduled for June 27th.

[heitorlessa]

Adding triage label to decide whether we make it ready for this Thursday's release

[leandrodamascena]

Unfortunately, it is not possible to release this feature tomorrow. I'm experiencing some issues with the Pydantic models for this implementation and need to figure out how to resolve them.

Rescheduling to the next release: 10/07/2024.

[leandrodamascena]

Updating this issue: We have a PR to add support for OpenAPI extensions. I need to improve the code and tests in this PR, but it is currently working and creating the schema as expexed:

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.event_handler.openapi.models import Server, APIKey, APIKeyIn

app = APIGatewayRestResolver(enable_validation=True)

servers = Server(
    url="http://example.com", 
    description="Example server", 
    openapi_extensions={"x-amazon-apigateway-endpoint-configuration": {"vpcEndpoint": "myendpointid"}} # SERVER LEVEL
    )


@app.get("/hello", openapi_extensions={"x-amazon-apigateway-integration": {"type": "aws", "uri": "my_lambda_arn"}}) # OPERATION LEVEL
def hello():
    return app.get_openapi_json_schema(
        servers=[servers], 
        security_schemes={
            "apikey": APIKey(
                    name="X-API-KEY",
                    description="API KeY", 
                    in_=APIKeyIn.header, 
                    openapi_extensions={"x-amazon-apigateway-authorizer": "custom"} # SECURITY SCHEME LEVEL
                )
            },
        openapi_extensions={"x-amazon-apigateway-gateway-responses": {"DEFAULT_4XX"}}) # ROOT LEVEL

def lambda_handler(event, context):
    return app.resolve(event, context)

openapi schema

{
   "openapi":"3.0.3",
   "info":{
      "title":"Powertools API",
      "version":"1.0.0"
   },
   "servers":[
      {
         "url":"http://example.com",
         "description":"Example server",
         "x-amazon-apigateway-endpoint-configuration":{
            "vpcEndpoint":"myendpointid"
         }
      }
   ],
   "paths":{
      "/hello":{
         "get":{
            "summary":"GET /hello",
            "operationId":"hello_hello_get",
            "responses":{
               "200":{
                  "description":"Successful Response",
                  "content":{
                     "application/json":{
                        
                     }
                  }
               },
               "422":{
                  "description":"Validation Error",
                  "content":{
                     "application/json":{
                        "schema":{
                           "$ref":"#/components/schemas/HTTPValidationError"
                        }
                     }
                  }
               }
            },
            "x-amazon-apigateway-integration":{
               "type":"aws",
               "uri":"my_lambda_arn"
            }
         }
      }
   },
   "components":{
      "schemas":{
         "HTTPValidationError":{
            "properties":{
               "detail":{
                  "items":{
                     "$ref":"#/components/schemas/ValidationError"
                  },
                  "type":"array",
                  "title":"Detail"
               }
            },
            "type":"object",
            "title":"HTTPValidationError"
         },
         "ValidationError":{
            "properties":{
               "loc":{
                  "items":{
                     "anyOf":[
                        {
                           "type":"string"
                        },
                        {
                           "type":"integer"
                        }
                     ]
                  },
                  "type":"array",
                  "title":"Location"
               },
               "type":{
                  "type":"string",
                  "title":"Error Type"
               }
            },
            "type":"object",
            "required":[
               "loc",
               "msg",
               "type"
            ],
            "title":"ValidationError"
         }
      },
      "securitySchemes":{
         "apikey":{
            "type":"apiKey",
            "description":"API KeY",
            "in":"header",
            "name":"X-API-KEY",
            "x-amazon-apigateway-authorizer":"custom"
         }
      }
   },
   "x-amazon-apigateway-gateway-responses":[
      "DEFAULT_4XX"
   ]
}

[leandrodamascena]

The pull request to add support for the OpenAPI extension feature is about to be merged, it's missing some last review - https://github.com/aws-powertools/powertools-lambda-python/pull/4703.

We've decided not to include it in the next release (11/07). Instead, we'll include it in the Powertools Python Alpha releases starting Friday (12/07) - https://pypi.org/project/aws-lambda-powertools/#history.

The reason we've decided not to add it in this release is that we prefer to allow customers some time to test it in an alpha release before deploying it to production environments. This is a sensitive addition to the OpenAPI feature, and that's why we've decided to introduce it as an alpha release.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

[leandrodamascena]

The OpenAPI extensions feature is available in this pre-release: https://pypi.org/project/aws-lambda-powertools/2.41.1a2/

If possible, test and report any type of bug or improvement before we go into production in 2 weeks.

[github-actions[bot]]

This is now released under 2.42.0 version!