TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference]

[christallire]

Hey guys, is there some reason I can't use same method name again?

For example, Let's say I have following grain

public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method();
}

and I've accidentally getting TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference] exception by changing signature like:

public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method(int accidentParameter);
}

so I hot-fixed the grain by adding a new signature like:

[Version(1)]
public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method();
    Task NewMethod(int accidentParameter);
}

and everything went okay. (this was a painful experience tho..)

The problem is I cannot use the same method name again and wonder why is that so. I've completely deployed version 1 grain and trying to reclaim the method name back but Orleans kept throwing TypeLoadException again like interface below:

[Version(2)]
public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method(int accidentParameter);
}

Here's my question:

1. Why am I not allowed to use the method name of the previous version?
2. Why am I not allowed to accept consequences (having side effects, being not backward compatible) but getting runtime errors instead of making silent signature changes?

I have a staging environment of 12 different hetero silos (because of microservices..) and only 2 of them got interface signature changes deployed. The one is the silo where grain got implemented and the one is the silo where we have the client (via GrainFactory) and the client is throwing TypeLoadException even if I have deployed version 2 of the grain interface on both silos.

[ReubenBond]

Hi @christallire, could you please clarify what triggers this issue for you? From your description, it sounds like defining this grain interface is enough:

public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method();
}

Is that correct?

[christallire]

Nope, need to have a different signature defined compared to no version(or version 0) like:

from:

[Version(1)]
public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method();
    Task NewMethod(int accidentParameter);
}

to:

[Version(2)]
public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method(int accidentParameter);
}

This is re: question 1

[christallire]

from

public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method();
}

to

public interface ISomeGrain : IGrainWithIntegerKey
{
    Task Method(int accidentParameter);
}

this is re: question 2

[ReubenBond]

Ok, so you've got a heterogeneous setup where you're modifying a grain interface and the problem is that it's no longer the same method, so Orleans doesn't know what to do with it. You can tell Orleans to treat them as the same method by giving the method an alias using [Alias("My method")] on the grain interface method. In that case maybe it will work. We should improve the error message either way.

[christallire]

But isn't that okay as long as agreed between both parties that are no longer the same method and both are aware of it? I was scratching my head because that doesn't just work.

Silo 1 knows: ISomeGrain Version 0 (just referencing defined assembly, not using the interface) Silo 2 knows: ISomeGrain Version 0 (just referencing defined assembly, not using the interface) Silo 3 knows: ISomeGrain Version 0 (just referencing defined assembly, not using the interface) Silo 4 knows: ISomeGrain Version 0 (just referencing defined assembly, not using the interface) Silo 5 knows: ISomeGrain Version 2 (the client, fully deployed) Silo 6 knows: ISomeGrain Version 2 (grain is activated here, fully deployed)

Silo 5 got TypeLoadException

Here's my question:

1. Why am I not allowed to use the method name of the previous version?
2. Why am I not allowed to accept consequences (having side effects, being not backward compatible) but getting runtime errors instead of making silent signature changes?

[ReubenBond]

You should be allowed to use a method name which was also defined in a previous version. Could you upload a repro? I'd like to investigate.

[YAJeff]

I'm running into this issue too. But only on 1 of the 3 replicas that my silo container is deployed to. Is there some kind of cache or short-term memory that is hanging onto these type definitions which can be deleted before a redeploy? This makes it challenging while just trying to develop a first release of an application.

[JCKortlang]

I am observing the same behavior in a non heterogeneous setup after changing a grain interface.

Interface:

//From
public interface IAssertionGrain : IGrainWithStringKey, IUpdateResourceNotifier
{
    Task<Assertion> Register(Assertion assertion);
    Task Delete();
    Task<Assertion?> GetDetails();
    Task<AssertionStatistics> QueryHistory(DateTimeOffset start, DateTimeOffset end);
}

//To
public interface IAssertionGrain : IGrainWithStringKey, IUpdateResourceNotifier
{
    Task<Assertion> Register(Assertion assertion);
    Task Delete();
    Task<Assertion?> GetDetails();
    Task<AssertionStatistics> QueryHistory(DateTimeOffset start, DateTimeOffset end, string gameRegion, string gameStage, string gameVersion = Metrics.Dimensions.DefaultValue);
}

Stack Trace:

An unhandled exception has occurred while executing the request.

System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Insights.Telemetry.Grains.Interfaces.IAssertionGrain,Insights.Telemetry.Grains.Interfaces],"1AC7528E")".

   at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 545

   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 223

   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77

   at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345

   at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72

   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 219

   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 201

   at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 129

   at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 125

   at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 347

--- End of stack trace from previous location ---

   at Orleans.Serialization.Invocation.ResponseCompletionSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 90

   at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 117

   at Orleans.Runtime.ActivityPropagationGrainCallFilter.Process(IGrainCallContext context, Activity activity) in /_/src/Orleans.Core/Diagnostics/ActivityPropagationGrainCallFilter.cs:line 75

   at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 88

   at Orleans.Runtime.GrainReferenceRuntime.InvokeMethodWithFiltersAsync[TResult](GrainReference reference, IInvokable request, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 74

   at Insights.Telemetry.SiloExtensions.TelemetrySiloExtensions.<>c.<<MapTelemetryRoutes>b__3_7>d.MoveNext() in /app/src/Telemetry/Insights.Telemetry.SiloExtensions/TelemetrySiloExtensions.cs:line 250

--- End of stack trace from previous location ---

   at Microsoft.AspNetCore.Http.RequestDelegateFactory.ExecuteTaskResult[T](Task`1 task, HttpContext httpContext)

   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)

   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)

   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)

   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|8_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

[ReubenBond]

Is the grain interface available on all hosts, @JCKortlang? That is a requirement today but it is a restriction which we are working on lifting, so that these kinds of errors will not occur in future.

[JCKortlang]

@ReubenBond it "should" be. We are deploying the latest container image across [1..n] containers.

Based on the AWS ECS logs, the most recent version of the image was deployed across all containers prior to the error. The issue was not reproducible when running the silo locally.

Note that I re-deployed the same code without changes and was unable to reproduce.

[ReubenBond]

Sorry, this is my fault for misreading. You are performing a rolling upgrade and are also modifying a grain interface method, which has changed is identity. If the method is still compatible, i.e, old versions of the code can ignore the new parameters, then you can specify the method id explicitly by applying the [Alias("MyMethodName")] attribute to it as I mentioned previously in this comment. I would recommend applying [Alias("x")] to all methods in general, since it reduces the chance for suprizes if you refactor type names and method signatures later. We should consider a best practices analyzer+codefix which automates this, similar to the [Id(x)] analyzer+codefix. EDIT: opened an issue here: https://github.com/dotnet/orleans/issues/8345

If the new method is not semantically equivalent to the old one (the new arguments cannot be safely ignored during the upgrade), then you should use Grain interface versioning to denote a breaking change. In this case, and in the case where you are adding a new method to the interface, you can use a two-phase upgrade: add the interface method in one deployment without using it anywhere, and then start using the method in a subsequent deployment.

I hope that helps to clarify things. We are still looking to improve the experience here for the next release.

[christallire]

I haven't had time to look this up for myself but:

I'm running into this issue too. But only on 1 of the 3 replicas that my silo container is deployed to. Is there some kind of cache or short-term memory that is hanging onto these type definitions which can be deleted before a redeploy? This makes it challenging while just trying to develop a first release of an application.

There's also an issue that Orleans doesn't allow breaking changes even with VersionAttribute like the quote above, especially with rolling updates, it just throws an exception if any of those signatures mismatches.

Preventing devs from making mistakes is a great idea, yet we really have to do something with this kind of runtime exception since we can't test it until it goes into production.

[sc-starman]

migration from 7.2.3 to 8.0.0-rc2 I got the same exception:

 Exception reading message Request [ sys.client/adf9fef2141c4881b1d4b8d5970cb3cb]->[S10.0.8.184:30000:0 sys.svc.manifest/10.0.8.184:30000@0] #302529 from remote endpoint 10.0.8.222:35278 to local endpoint 10.0.8.184:30000
System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Orleans.Runtime.IClusterManifestSystemTarget,Orleans.Core],"4EFCA109")".
   at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 545
   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 223
   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
   at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345
   at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72
   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169
   at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
   at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 349

[sc-starman]

migration from 7.2.3 to 8.0.0-rc2 I got the same exception:

 Exception reading message Request [ sys.client/adf9fef2141c4881b1d4b8d5970cb3cb]->[S10.0.8.184:30000:0 sys.svc.manifest/10.0.8.184:30000@0] #302529 from remote endpoint 10.0.8.222:35278 to local endpoint 10.0.8.184:30000
System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Orleans.Runtime.IClusterManifestSystemTarget,Orleans.Core],"4EFCA109")".
   at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 545
   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 223
   at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter`1.ApplyInner(TypeSpec input, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
   at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in /_/src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345
   at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader`1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72
   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader`1& reader, SchemaType schemaType) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
   at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader`1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169
   at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader`1& reader) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
   at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /_/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 349

It was my mistake that I didn't update the client to 8.0.0-rc2, after update everything seems to be fine again.

[ReubenBond]

Thank you for confirming, @sc-starman