feat: Add sam generate openapi command

Open dcabib opened this issue 2 months ago • 1 comments

GitHub Issue #1473 Comment - OpenAPI Compliance Discussion

Hey SAM CLI team! 👋

I've implemented sam generate openapi for issue #1473 - it's working and tested (94% coverage, 5,956 tests passing, verified on complex production templates). But I need architectural guidance on 5 OpenAPI compliance decisions before opening the PR.

What Works Today

Command successfully extracts OpenAPI from SAM templates via SAM Translator:

sam generate openapi -t template.yaml --openapi-version 3.0 -o api.yaml

Tested on production template: 35 API routes, 6 Lambda functions, Cognito authorization - all generated correctly with proper CORS and security configuration.

The 5 Compliance Gaps (Need Community Direction)

1. CloudFormation Intrinsic Functions

Current Output:

info:
  title: {Ref: AWS::StackName}
paths:
  /users:
    get:
      x-amazon-apigateway-integration:
        uri: {Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${UserFunction.Arn}/invocations}
security:
  - CognitoAuth:
      providerARNs: [{Ref: UserPoolArn}]

OpenAPI Standard: Plain resolved values

info:
  title: "My API"
paths:
  /users:
    get:
      # No CloudFormation syntax

Why It Happens: SAM Translator preserves template structure - doesn't resolve CloudFormation parameters or functions.

The Problem: CloudFormation refs make the spec less usable for swagger-codegen and other OpenAPI tooling.

Options:

A) Keep CloudFormation refs as-is (shows template structure, matches SAM deployment spec)
B) Resolve all refs using --parameter-overrides (cleaner output, but requires all parameters be provided)
C) Add --resolve-refs flag for user choice (default: keep refs)

My Recommendation: Option C - Default to raw output (accurate to SAM), add flag for resolved version if needed.

2. AWS-Specific Extensions

Current Output:

paths:
  /users/{userId}:
    get:
      x-amazon-apigateway-integration:
        httpMethod: POST
        type: aws_proxy
        uri: {Fn::Sub: arn:aws:apigateway:...}
      x-amazon-apigateway-auth:
        type: cognito_user_pools

x-amazon-apigateway-gateway-responses:
  DEFAULT_4XX:
    responseParameters:
      gatewayresponse.header.Access-Control-Allow-Origin: "'*'"

OpenAPI Standard: No vendor-specific extensions

Why It Happens: These ARE the actual API Gateway deployment configurations that SAM generates. They show exactly how the API will be deployed.

The Problem: Pure OpenAPI specs shouldn't have vendor extensions.

Options:

A) Keep AWS extensions (shows real deployment details, valuable for infrastructure docs)
B) Strip all x-amazon-apigateway-* fields (cleaner for generic tooling)
C) Add --strip-aws-extensions flag for choice

My Recommendation: Option A - Keep extensions. Reasons:

They're accurate and valuable for understanding deployment
Most OpenAPI tools (including swagger-codegen) ignore unknown extensions
Removing them loses important configuration information
If someone needs pure OpenAPI, they can post-process

3. Swagger 2.0 vs OpenAPI 3.0

Current Output: Swagger 2.0 format (what SAM Translator generates)

swagger: '2.0'
securityDefinitions:  # Swagger 2.0 location
  CognitoAuth: {...}

Why: SAM Translator hasn't migrated to OpenAPI 3.0 yet - still outputs Swagger 2.0.

Current Implementation: Added --openapi-version flag

--openapi-version 2.0: Raw SAM Translator output (Swagger 2.0)
--openapi-version 3.0: Auto-converts to OpenAPI 3.0 (moves securityDefinitions → components.securitySchemes)

Default: OpenAPI 3.0 (modern standard)

Question: Is defaulting to 3.0 correct, or should we default to 2.0 (matches SAM Translator output)?

My Recommendation: Keep default as 3.0 - Most tooling prefers OpenAPI 3.0. Users can specify 2.0 if needed.

4. Missing `servers` Section

Current Output: No servers array

OpenAPI 3.0 Requirement:

servers:
  - url: https://api.example.com/v1
    description: Production API

Why Missing: API Gateway assigns URL at deployment time. Pre-deployment, the actual URL doesn't exist yet.

The Problem:

OpenAPI 3.0 spec technically requires servers section
But we can't know the URL from just the SAM template
URL format: https://{apiId}.execute-api.{region}.amazonaws.com/{stage}
Where {apiId} is assigned by CloudFormation at deploy time

Options:

A) Omit entirely (current) - Spec is technically invalid per OpenAPI 3.0
B) Add placeholder: https://example.com/REPLACE_AFTER_DEPLOYMENT
C) Add template URL: https://{apiId}.execute-api.{region}.amazonaws.com/{stage} with note
D) Only include if user provides --api-url parameter

My Recommendation: Option C - Add template URL with CloudFormation-style variables. Shows the structure, users can substitute {apiId} and {stage} after deployment. Example:

servers:
  - url: "https://{apiId}.execute-api.{region}.amazonaws.com/{stage}"
    description: API Gateway endpoint (replace {apiId} and {stage} after deployment)
    variables:
      apiId: {description: "API Gateway REST API ID", default: "REPLACE_ME"}
      stage: {description: "Deployment stage", default: "dev"}

5. Missing Request/Response Schemas

Current Output:

paths:
  /users/{userId}:
    get:
      responses: {}  # Completely empty!
    post:
      responses: {}  # No request body schema either

OpenAPI Standard: Should define detailed schemas

paths:
  /users/{userId}:
    get:
      parameters:
        - name: userId
          in: path
          required: true
          schema:
            type: string
            pattern: '^[0-9]+$'
      responses:
        200:
          description: User found
          content:
            application/json:
              schema:
                type: object
                properties:
                  userId: {type: string}
                  name: {type: string}
                  email: {type: string, format: email}
        404:
          description: User not found
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                name: {type: string}

Why Missing: SAM templates define ROUTES but not DATA MODELS. Example SAM template:

Resources:
  UserFunction:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        CreateUser:
          Type: Api
          Properties:
            Path: /users
            Method: post
            # NO RequestModel or ResponseModel defined!

SAM supports RequestModel/ResponseModel, but most templates don't use them.

The Real Problem: This is a SAM template limitation, not a command limitation. If schemas aren't in the template, we can't generate them.

Options:

A) Leave empty (current) - Accurate to template, documents limitation
B) Generate generic placeholder schemas (type: object)
C) Require users to add RequestModel/ResponseModel to SAM template if they want schemas
D) Add --infer-schemas flag to attempt inference from Lambda function type hints (complex, fragile)

My Recommendation: Option A - Leave empty and document. Reasons:

Being accurate to template is better than making up schemas
Users needing schemas should define models in SAM template
swagger-codegen works fine without schemas (generates generic types)
Option D (inference) is scope creep and error-prone

Summary of Recommendations

Gap	Proposed Solution	Reasoning
CloudFormation refs	Keep, add `--resolve-refs` flag later if needed	Accuracy first, convenience optional
AWS extensions	Keep all extensions	Valuable config info, ignored by tools anyway
Swagger vs OpenAPI	Default to 3.0, flag for 2.0 ✅	Modern standard preferred
servers section	Add template URL with variables	Shows structure, user substitutes after deploy
Schemas	Leave empty, document	Can't generate what's not in template

The Trade-Off

Current approach: Generates "deployment-faithful" OpenAPI

Shows what SAM actually creates
Includes AWS config
Has CloudFormation template structure

Alternative: "tooling-friendly" OpenAPI

Resolved values
No AWS stuff
Pure standard compliance
Less accurate to actual deployment

Question: Which philosophy should SAM CLI adopt? The original issue says "use swagger-codegen" but doesn't specify which approach.

What I'm Asking

Agree/disagree with the 5 recommendations above?
Should we do this iteratively? (MVP now, enhancements based on feedback)
Any other compliance concerns I missed?

Branch is ready: https://github.com/dcabib/aws-sam-cli/tree/feat/generate-openapi-command
(Blocked by AWS security hook on unrelated file, but code is committed locally)

Happy to adjust based on team direction! 🚀

Oct 01 '25 23:10 dcabib

Which issue(s) does this change fix?

Fixes #1473

Why is this change necessary?

SAM CLI currently generates OpenAPI/Swagger documents automatically at deploy time, but there's no way to access this generated document during the build process. This creates challenges for development teams who want to:

Generate API client code using swagger-codegen or OpenAPI Generator
Create API documentation during CI/CD pipelines
Integrate with other OpenAPI-based tooling and workflows
Validate API contracts before deployment

Currently, teams must hand-manage OpenAPI documents separately instead of leveraging SAM's automatic generation capabilities, leading to:

Duplicated effort maintaining both SAM templates and OpenAPI specs
Potential inconsistencies between deployed APIs and documentation
Inability to use modern API development workflows that rely on OpenAPI specs

This feature has been requested by the community for over 6 years with strong support from multiple users.

How does it address the issue?

This PR implements a new sam generate openapi command that extracts OpenAPI specifications from SAM templates using the existing SAM Translator library. The implementation includes:

Command Structure

Adds new generate command group to support future artifact generation
Implements openapi subcommand with comprehensive options
Usage: sam generate openapi -t template.yaml --openapi-version 3.0 -o api.yaml

Key Features

Automatic API Detection: Auto-detects single API resources in templates
Multi-API Support: Supports --api-logical-id flag for templates with multiple APIs
Format Options: Outputs YAML (default) or JSON via --format flag
Version Support: Generates OpenAPI 3.0 (default) or Swagger 2.0 via --openapi-version flag
CloudFormation Integration: Preserves CloudFormation intrinsic functions and parameter references
AWS Extensions: Maintains x-amazon-apigateway-* extensions showing actual deployment configuration
Flexible Output: Supports stdout or file output via --output-file flag

Architecture

Core generation logic in samcli/lib/generate/openapi_generator.py using SAM Translator
Swagger 2.0 to OpenAPI 3.0 conversion in samcli/lib/generate/openapi_converter.py
CLI command implementation following SAM CLI patterns with context managers
Comprehensive exception handling with user-friendly error messages

Implementation Philosophy

The command generates "deployment-faithful" OpenAPI documents that accurately reflect what SAM creates at deploy time:

Preserves CloudFormation syntax (can be resolved with --parameter-overrides)
Includes AWS-specific extensions (valuable for infrastructure documentation, ignored by most tools)
Defaults to modern OpenAPI 3.0 standard
Leaves schemas empty if not defined in template (accurate vs. synthetic)

What side effects does this change have?

Positive Side Effects

Enables modern API-first development workflows with SAM
Reduces maintenance burden by avoiding duplicate OpenAPI specs
Improves API documentation accuracy by generating directly from source templates
Enables integration with ecosystem tools (swagger-codegen, OpenAPI Generator, etc.)

Potential Considerations

CloudFormation References: Generated OpenAPI contains CloudFormation intrinsic functions (e.g., {Ref: AWS::StackName}). This is intentional for accuracy but may require post-processing for some tooling. Can be resolved with --parameter-overrides.
AWS Extensions: Output includes x-amazon-apigateway-* extensions. These are ignored by most OpenAPI tools but provide valuable deployment configuration details.
Missing Schemas: If SAM templates don't define RequestModel/ResponseModel, the generated OpenAPI will have empty response objects. This is accurate to the template but may require users to enhance their SAM templates for complete documentation.
No Breaking Changes: This is a purely additive feature with no changes to existing SAM CLI functionality.

Mandatory Checklist

[x] Add input/output type hints to new functions/methods: All new functions have complete type hints (str, Optional, Dict, etc.)
[x] Write design document if needed: Not required - straightforward feature addition following existing SAM CLI patterns. Leverages existing SAM Translator library.
[x] Write/update unit tests:
- Complete unit test coverage for all modules
- Tests for command group, OpenAPI command, context, exceptions
- Tests for OpenAPI generator and converter logic
- Coverage: 94.05% (meets 94% requirement)
[x] Write/update integration tests:
- Integration tests for command execution
- Tests with sample SAM templates
- Tests for multiple scenarios (single API, multiple APIs, error cases)
- Located in tests/integration/generate/openapi/
[x] Write/update functional tests if needed: Integration tests serve as functional tests for this feature
[x] make pr passes:
- ✅ All 5,957 tests pass
- ✅ Coverage requirement met (94.05%)
- ✅ Black formatting applied
- ✅ Type hints added
[x] make update-reproducible-reqs if dependencies were changed: No new dependencies added - uses existing SAM Translator and Click libraries
[x] Write documentation:
- Comprehensive docstrings for all modules, classes, and functions
- Command help text and descriptions
- Usage examples in command help
- Additional documentation can be added to official docs upon approval

Test Results

Required test coverage of 94% reached. Total coverage: 94.05%
====================== 5957 passed, 21 skipped in 52.31s =======================

Testing on Production Templates

Successfully tested on complex production template with:

35 API routes
6 Lambda functions
Cognito authorization
CORS configuration
All generated correctly with proper security and integration settings

Open Questions for Maintainers

While the implementation is complete and tested, I'd appreciate guidance on these architectural decisions before finalizing:

CloudFormation References: Current approach preserves refs for accuracy. Should we add --resolve-refs flag in future PR?
Default OpenAPI Version: Currently defaults to 3.0 (modern standard). Should we default to 2.0 (matches SAM Translator output)?
servers Section: Currently omitted (URL unknown pre-deployment). Should we add template URL with variables?

Happy to adjust based on team preferences. The current implementation follows a "deployment-faithful" philosophy but can be enhanced iteratively based on user feedback.

Example Usage

# Basic usage - output to stdout
sam generate openapi -t template.yaml

# Save to file with specific format
sam generate openapi -t template.yaml -o api.yaml --format yaml

# Generate Swagger 2.0 instead of OpenAPI 3.0
sam generate openapi -t template.yaml --openapi-version 2.0

# Specify API for templates with multiple APIs
sam generate openapi -t template.yaml --api-logical-id MyApi

# JSON output to stdout
sam generate openapi -t template.yaml --format json

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Oct 08 '25 13:10 dcabib

feat: Add sam generate openapi command

GitHub Issue #1473 Comment - OpenAPI Compliance Discussion

What Works Today

The 5 Compliance Gaps (Need Community Direction)

1. CloudFormation Intrinsic Functions

2. AWS-Specific Extensions

3. Swagger 2.0 vs OpenAPI 3.0

4. Missing servers Section

5. Missing Request/Response Schemas

Summary of Recommendations

The Trade-Off

What I'm Asking

Which issue(s) does this change fix?

Why is this change necessary?

How does it address the issue?

Command Structure

Key Features

Architecture

Implementation Philosophy

What side effects does this change have?

Positive Side Effects

Potential Considerations

Mandatory Checklist

Test Results

Testing on Production Templates

Open Questions for Maintainers

Example Usage

4. Missing `servers` Section