Remote Schema Merging/Routing Issue with Unique Namespaces in Hasura 2.44

[doudian719]

Description

I'm encountering an unexpected behavior with Hasura version 2.44 when registering multiple remote schemas originating from different Kubernetes namespaces. I have two remote schemas configured:

Namespace A: Contains a remote API registered with a unique root field namespace and a unique prefix.

Namespace B: An exact copy of Namespace A's API, but deployed under a different Kubernetes namespace with its own unique root field namespace and prefix.

Steps to Reproduce

1. Configure Remote Schemas:

Register both remote schemas in Hasura with their respective GraphQL server URLs.

For each schema, set a unique prefix and root field namespace that includes the respective Kubernetes namespace information.

Enable the “Forward client headers” option.

2. Test the APIs:

Use Postman (or another GraphQL client) to send queries to the Hasura endpoint.

Query a field that exists only in the Namespace B API (an extra field added specifically in Namespace B to help identify the routing).

Observe that the query intended for Namespace B returns an error indicating the field is missing—signifying that the request is actually routed to Namespace A's API.

3. Observations:

An introspection query against Hasura’s merged schema shows that Namespace B's definitions are mixed with those from Namespace A.

When the Namespace A remote schema is removed from the Hasura console, the Namespace B API works as expected without any routing issues.

Expected Behavior

Each remote schema should be isolated using its root field namespace and unique type prefixes. This isolation should ensure that:

Queries intended for Namespace B are routed to its corresponding backend service.

There is no overlapping or merging of the GraphQL type definitions from Namespace A into Namespace B’s schema.

Actual Behavior

When both remote schemas are registered:

The introspection result shows that Namespace B's schema includes definitions from Namespace A.

Queries targeting Namespace B are being routed to Namespace A’s backend service, resulting in errors (e.g., missing fields unique to Namespace B).

Additional Context

Both remote schemas have their GraphQL Server URLs correctly configured.

The schemas are almost identical, with the only difference being an extra field in Namespace B’s schema for testing.

It appears that when remote schemas have similar type definitions, Hasura's stitching logic might be merging them—even with unique prefixes and namespaces set.

The “Forward client headers” option is enabled, although this should not impact routing decisions.

Questions

Is this behavior (i.e., merging definitions or misrouting between remote schemas with similar structures) intended, or is it a known bug in Hasura 2.44?

If it is a bug, are there any recommended workarounds or configuration changes (beyond using unique prefixes/namespaces) to properly isolate remote schema definitions?

Any insights or suggestions on how to avoid this schema merging/routing conflict would be greatly appreciated.

Thanks in advance for your assistance!

[robertjdominguez]

Hey, thanks for the detailed report—this is super helpful, @doudian719 🙏

What you’re seeing isn’t expected behavior. With unique root field namespaces and type prefixes in place, Hasura should be isolating those remote schemas completely. If the merged schema is showing overlapping types or routing queries incorrectly, especially when those fields are only present in one of the backends, that sounds like a bug in the schema stitching logic.

A couple things to try or confirm for us:

1. Double-check that both type prefixes and root field namespaces are unique and consistently applied across all types and root fields in each remote schema.

2. If the schemas are truly identical (outside of one extra field), you might try customizing the type names further (e.g. UserA vs UserB) to see if that helps force separation.

But even with that, Hasura shouldn’t be merging types across two remote schemas with different prefixes. The fact that removing Namespace A resolves the issue for Namespace B confirms it’s likely an internal conflict during stitching.

I’ll flag this internally to get eyes on it, but in the meantime, can you share your remote_schemas config block (with sensitive stuff redacted) and maybe the result of the merged introspection query? That’ll help us reproduce it more directly.

Appreciate you surfacing this.

[doudian719]

Hi @robertjdominguez, thank you so much for your prompt reply and for taking the time to look into this issue.

Regarding your suggestion to double-check that both type prefixes and root field namespaces are unique and consistently applied across all types and root fields in each remote schema—I can confirm that they are indeed unique. We have been running Hasura in our environment for over a year without any issues, and our instance is currently hosting over 80 APIs. This schema merging problem has only started occurring recently.

To provide a bit more context:

1. Our remote_schemas configurations are all created via API calls to Hasura. This automated process ensures that each registration has a unique Root Field Namespace, as well as unique type prefixes and suffixes that combine the API’s own name with its Kubernetes namespace. We specifically designed it this way to avoid any potential conflicts.

2. When we use Postman to send an introspection query (IntrospectionQuery) to the Hasura server, the response clearly shows that the Namespace B API is referencing an INPUT_OBJECT from Namespace A.

3. Additionally, the fact that removing the remote schema for Namespace A causes Namespace B to function correctly further suggests that the stitching logic might be mistakenly merging types between these two schemas.

Thank you again for your support and assistance. I truly appreciate your help and look forward to any further insights you might have.

[robertjdominguez]

Thanks for confirming all of that, @doudian719 — really appreciate the thorough follow-up.

Based on what you’ve shared (especially the INPUT_OBJECT from Namespace A showing up in Namespace B’s schema), this definitely points to something going wrong with the type isolation during remote schema stitching. Even with unique type prefixes, root field namespaces, and a clean automated registration flow, Hasura should not be merging types across remotes like this — so this isn’t expected behavior.

Internally, I believe the stitching logic is meant to treat prefixed types as separate, even if the underlying structure is identical. But what you’re running into suggests a deeper bug that kicks in when types have identical structures and share a common ancestry.

As a temporary workaround, can you experiment with renaming key input object types in the source schema (even with prefixing already in place) just to help Hasura treat them more distinctly — not ideal, but potentially unblocks you while we dig in.

I’ll escalate this internally. Thanks again for surfacing such a clear report — this helps a lot.

[greglboxer]

I think we are also coming across this issue in our usage of remote schemas (v2.44, although we have seen it in an earlier version).

We have an "okrs" and "roadmaps" Hasura server each sitting on top of Postgres databases.

We reference both of these in another Hasura instance with the remote schema feature. Remote schema types are prefixed with 'okrs_' and 'roadmaps_' to avoid conflicts.

Unfortunately the stitching seems to be getting confused when trying to use them as query variables. objectives is a tracked table in the okrs server.

[alex4506]

We also encountered a similar problem. When registering the same type with multiple remote schemas, the type name of the remote schema that registers first appears in the returns of subsequent other remote schemas, and methods such as renaming cannot solve it.

[nishchhal-shrestha]

We are also experiencing exact same problem as https://github.com/hasura/graphql-engine/issues/10717#issuecomment-2808113639

We have been using hasura for past 2 years and this issue suddenly came. I'm using version 2.35.0. We have not upgraded hasura version. This was working properly previously.

[manuelbernhardt]

We experienced the same issue, seemingly out of nowhere. Without any changes on our end, the production, staging and local deployments started failing. We solved this for now by aliasing some types but have still absolutely no idea why this suddenly started happening. @robertjdominguez is there any caching / cache eviction in place for remote schema stitching which could explain why this happens at runtime without change in the deployments?

[robertjdominguez]

Thanks for adding instances of this, folks. I've re-raised it internally and hope to get back to you soon!

[jberryman]

@manuelbernhardt can you give a little more detail:

• the version of hasura was consistent throughout (working and nonworking)?
• did the problem happen after a restart or fresh instance was launched?
• did it start after a metadata change of any kind?

[manuelbernhardt]

@jberryman

• the Hasura version deployed in both production and staging environments was / is v2.37.1. we also tried locally in development with the latest version at that time (2.48.x, I'm unsure about which one of the minor versions, @timoransky maybe you know?)
• the problem appeared seemingly out of nowhere. there was no restart / container redeployment. Asset resolution just stopped working
• metadata wasn't changed on the running instance. in our setup, we also redeploy the Hasura container after a metadata update so I'm confident this wasn't the case

The behavior really baffled us because it started happening on running instances without external changes of any kind. Once the behavior started, none of the deployments worked (staging, production, locally), independently of the Hasura version used.

[nishchhal-shrestha]

Do we have any ETA on this issue? We are depending on the Hasura for our various reporting modules

[jberryman]

A fix was just merged and should make it into the next 2.48 release

[nishchhal-shrestha]

@jberryman does this https://github.com/hasura/graphql-engine/releases/tag/v2.48.6 release fix this issue?

[jberryman]

@nishchhal-shrestha yes, that's the second changelog entry. Let us know if you still experience any issues