modelcontextprotocol
Add end to end testing for Oauth flow
Problem: some things like the Oauth flow in inspector are challenging to test either manually or with unit tests, and need some form of integration and/or regression tests to ensure a stable experience.
Potential solutions: browser-based testing tools such as Playwright or Puppeteer could be a good fit for these types of tests.
Scope of this enhancement issue is to cover testing basic Oauth functionality in Inspector, since this has been one of the trickier areas. If this works well and proves to be maintainable we can add similar tests for other e2e scenarios.
Resources:
- https://www.testim.io/blog/how-to-test-oauth-authentication/ is a general guide on testing Oauth
- Recommendation from Max: AS- Keycloak has a docker container that could be useful for local testing. They have a feature where you can export or import the entire application state to a JSON file - so we just check in the JSON and then in CI whenever you spin up the server you can import it fresh. Makes it easy to have a repeatable set of clients / users.
- This server has been useful when doing manual testing: https://github.com/Azure-Samples/mcp-auth-servers/tree/main/src/github-app-session
Consider testing using the updated version in Max's PR: https://github.com/modelcontextprotocol/inspector/pull/279
BTW, @max-stytch on this topic in Discord:
If we want to test lack of conformance - e.g. if an AS does something wrong and we want to confirm the Inspector behaves appropriately, or if we want much more pluggable AS behaviors to fully excercise the inspector, I don't think Keycloak is a great fit as a test dependency. The whole point of Keycloak is that it works and works well, so it can really only be used for happy path stuff. If we want to test a specific behavior (e.g. falling back to openid metadata if as metadata is unavailable) it is a lot easier to write a lot of little AS implementations than it is to fight keycloak config to create whatever weird state we need.
This is definitely something that would be super useful. I hear lots of demand. Might also be worth merging with #694 where an early attempt to this has been made.
Closing as not planned and labeling with v2 in case we want to review for next version.