Custom EventBridge Schema Registry - Cannot export JSONSchemaDraft4 for non discovered or non aws managed schemas

[frenayj]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

Description

When setting up the generator with a custom registryName (other than discovered-schemas), running npm run generate will yield the following exception: ForbiddenException: You cannot export non discovered or non aws managed schemas.

This seems to be due to the usage of exportSchema() on every returned schema by default (verified with Python boto3.schemas lib too).

We might need to make this feature optional or add a condition on the registryType.

Steps to reproduce

1. Create a custom EventBridge Schema Registry
2. Create a new Schema in your new EventBridge Schema Registry
3. Set the generator to point to the custom EventBridge Schema Registry (registryName)

generators: [
    [
      '@eventcatalog/plugin-doc-generator-amazon-eventbridge',
      {
        eventBusName: 'MyEventBus',
        region: 'eu-west-2',
        registryName: 'MySchemaRegistry',
        schemaTypeToRenderToEvent: 'OpenAPI',
        credentials: {
          accessKeyId: process.env.AWS_ACCESS_KEY,
          secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        },
      },
    ],
  ]

4. Run npm run generate

Expected behavior

As for the default discovered-schemas registry, we expect listed schemas to be mapped into new events/ directories containing the templated markdown docs.

In the case of a custom EventBridge Schema Registry, the documentation could be templated without exporting the JSONSchemaDraft4 schema files.

Actual behavior

a 403 ForbiddenException: You cannot export non discovered or non aws managed schemas exception is thrown.

Generating EventCatalog docs using: @eventcatalog/plugin-doc-generator-amazon-eventbridge
/Users/john/my-event-catalog/node_modules/@aws-sdk/client-schemas/dist-cjs/protocols/Aws_restJson1.js:2888
    const exception = new models_0_1.ForbiddenException({
                      ^

ForbiddenException: You cannot export non discovered or non aws managed schemas.
    at deserializeAws_restJson1ForbiddenExceptionResponse (/Users/john/my-event-catalog/node_modules/@aws-sdk/client-schemas/dist-cjs/protocols/Aws_restJson1.js:2888:23)
    at deserializeAws_restJson1ExportSchemaCommandError (/Users/john/my-event-catalog/node_modules/@aws-sdk/client-schemas/dist-cjs/protocols/Aws_restJson1.js:1815:25)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /Users/john/my-event-catalog/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /Users/john/my-event-catalog/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:11:20
    at async StandardRetryStrategy.retry (/Users/john/my-event-catalog/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
    at async /Users/john/my-event-catalog/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
    at async /Users/john/my-event-catalog/node_modules/@eventcatalog/plugin-doc-generator-amazon-eventbridge/lib/lib/aws.js:12:30
    at async Promise.all (index 0)
    at async exports.default (/Users/john/my-event-catalog/node_modules/@eventcatalog/plugin-doc-generator-amazon-eventbridge/lib/index.js:38:21) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 403,
    requestId: '4526e20b-4fbd-44a9-9a72-4bd4d427e791',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Code: 'Forbidden'
}
Error: Command failed: cross-env PROJECT_DIR=/Users/john/my-event-catalog npm run generate
    at checkExecSyncError (node:child_process:826:11)
    at execSync (node:child_process:900:15)
    at Command.<anonymous> (/Users/john/my-event-catalog/node_modules/@eventcatalog/core/bin/eventcatalog.js:108:5)
    at Command.listener [as _actionHandler] (/Users/john/my-event-catalog/node_modules/commander/index.js:922:31)
    at Command._parseCommand (/Users/john/my-event-catalog/node_modules/commander/index.js:1503:14)
    at Command._dispatchSubcommand (/Users/john/my-event-catalog/node_modules/commander/index.js:1443:18)
    at Command._parseCommand (/Users/john/my-event-catalog/node_modules/commander/index.js:1460:12)
    at Command.parse (/Users/john/my-event-catalog/node_modules/commander/index.js:1292:10)
    at run (/Users/john/my-event-catalog/node_modules/@eventcatalog/core/bin/eventcatalog.js:115:7)
    at Object.<anonymous> (/Users/john/my-event-catalog/node_modules/@eventcatalog/core/bin/eventcatalog.js:122:1) {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 36420,
  stdout: null,
  stderr: null
}

Your environment

• EventCatalog version used: 0.0.6
• Environment name and version: Chrome 100.0.4896.127, Node.js 16.13.2
• Operating system and version: MacOS 12.3.1

[boyney123]

Hey @frenayj ,

Yeah this is very interesting indeed! Will take a look around and explore abit more, but as you said if we cannot get it I think having some condition in there would be good too. Guess would mean less hydration for you, but would make things work...

[boyney123]

Hey @frenayj

I know with custom registry you can upload OpenAPI files or JSONSchema files, which are you using out of interest?

Are you uploading OpenAPI files?

[frenayj]

Hey @frenayj

I know with custom registry you can upload OpenAPI files or JSONSchema files, which are you using out of interest?

Are you uploading OpenAPI files? Hey 👋

We are uploading OpenAPI files for now but switching to JSONSchemaDraft4 isn't really a problem as long as the doc-gen lets us "discover" schemas from a "custom Schema Registry".

For now it looks like AWS only let us "download" OpenAPI schemas from a custom Schema Registry:

See the same Action button on the discovered-schemas registry:

🙂 your colleages at AWS might know why this limitation still exists

[boyney123]

Hey @frenayj ,

Yeah interesting, I think at the moment we will have to add some checks in generator so we can get that working.

I'm also talking to the team and passing on the feedback 👍

I'm happy to try and make the change to skip it, unless you wanted to have a go? Let me know if you have the time, otherwise I should be able to pick it up

Thanks for raising it and providing the feedback 🙏

[frenayj]

tbh I had a quick look a few weeks ago but I struggled with too many deps conflicts while building the project and gave up 😬

[boyney123]

Ah o no @frenayj , any idea what kind of issues you were seeing?

[adlecluse]

I'm also wondering why AWS wouldn't allow to export a custom schema? 🤔

Tried it with the CLI and it gives the same error message:

aws schemas export-schema --registry-name MyRegistry --schema-name MyOwnEvent --type JSONSchemaDraft4

Results in

An error occurred (ForbiddenException) when calling the ExportSchema operation: You cannot export non discovered or non aws managed schemas.

[boyney123]

Thanks @adlecluse , let me find out more details on this, I will ask the EventBridge team.

[boyney123]

@adlecluse so I spoke with PM on the EventBridge team, looks like this feature missed the initial launch of the schema registry and a feature request has been launched internal for it.

Thanks for providing the feedback, if you would like to keep updated or connect with the PM directly with the issue (any anyone else reading this), feel free to reach out to me and I can connect you. Think it would be great to understand your usecase for the feature

[adlecluse]

@adlecluse so I spoke with PM on the EventBridge team, looks like this feature missed the initial launch of the schema registry and a feature request has been launched internal for it.

Thanks for providing the feedback, if you would like to keep updated or connect with the PM directly with the issue (any anyone else reading this), feel free to reach out to me and I can connect you. Think it would be great to understand your usecase for the feature

Thanks for the update @boyney123!

For me this was for experimenting with EventBridge and the EventCatalog and see if it fits my project. The main reason at first was that using custom schemas is actually free of charge as I read it in the pricing docs

Usage of the Schema Registry for AWS and custom schemas is free.

While the schema discovery has an actual cost, not much, but still something to consider when choosing a specific implementation path.

Another reason that I would use custom schemas is that I have my agreed upon JSON schemas for my events created upfront before the event will be created and used. So for me this looked the appropriate place to put them and I expected them to pop up in the EventCatalog.

But good to know there is a feature request ongoing for this at the EventBridge team 👍

[boyney123]

@adlecluse thanks for putting in the fix for now, appreciate it. Going to close this issue, and when I hear about some support for custom schemas, we can relook, hope that's OK.