Consider enabling the interpreter by default for release builds

[rolfbjarne]

Description

Currently the interpreter is enabled for Debug builds:

https://github.com/dotnet/maui/blob/6b43399d3f1b9520925bebdab680076787ab5e60/.nuspec/Microsoft.Maui.Controls.props#L8-L9

At least for iOS and Mac Catalyst, we might want to make this the default for Release builds as well, because I'm seeing a lot of customers running into problems where their app crashes once they switch to Release builds, because they used some feature in their app that requires the interpreter.

Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.

While the interpreter might slow things down, the performance will be fine for most apps, and if someone really needs the additional performance they can disable the interpreter.

The main downside might be that crash reports will be rather useless, because all managed stack frames will just show up as the interpreter doing "stuff".

Steps to Reproduce

Triage bugs for a while.

Alternatively read about customer's experiences online (Twitter, blog posts, etc.).

Example blog post: https://www.andreasnesheim.no/my-experience-with-migrating-my-app-from-xamarin-forms-to-net-maui/

Link to public reproduction project repository

N/A

Version with bug

7.0 (current)

Last version that worked well

Unknown/Other

Affected platforms

iOS, macOS

Affected platform versions

All so far

Did you find any workaround?

1. Better documentation.
2. Better error messages.

Relevant log output

No response

References

• https://github.com/dotnet/maui/issues/13016
• https://github.com/dotnet/maui/issues/9176
• https://github.com/dotnet/maui/issues/12537
• https://github.com/dotnet/runtime/issues/74015
• https://github.com/dotnet/maui/issues/12316
• https://github.com/dotnet/maui/issues/12519
• https://github.com/dotnet/maui/issues?q=is%3Aissue+sort%3Aupdated-desc+useinterpreter+release+
• https://github.com/dotnet/maui/issues/18870

[symbiogenesis]

Is there a list of known deficiencies in the current release mode tooling? I have experienced similar issues.

[noque-lind]

I would like to know which features require the Interpreter. Because we have run into the same thing. UseIntepreter works for release mode, otherwise it crashes at launch. I'd like to try the app without the interpreter because our client is not really happy with the performance compared to how the app ran in Xamarin Forms.

[Ghevi]

@noque-lind @symbiogenesis Can't provide a list but i know that Entity Framework Core requires the interpreter on iOS 9176, 12537

[mattleibow]

@jonathanpeppers this feels like a nice way to make customers happy and still have all the power to get that performance?

@rolfbjarne do we know what part of entity framework needs the interpreter? Technically we can make all platforms use a slower code path because a popular library needs it. However, maybe entity framework can provide an alternate code path for AOT-only environments?

[jonathanpeppers]

@mattleibow just don't do this for Android, because it disables the JIT. iOS/MacCatalyst can't JIT, so the only downside is whatever the increase to app size. Plus there might be some things about weird stack traces Rolf mentions above.

[rolfbjarne]

@mattleibow

@rolfbjarne do we know what part of entity framework needs the interpreter? Technically we can make all platforms use a slower code path because a popular library needs it. However, maybe entity framework can provide an alternate code path for AOT-only environments?

I haven't investigated why entity framework needs the interpreter. I also believe there are other libraries that needs it, so it's not only entity framework.

[symbiogenesis]

Linking this bug for context. It is what I have marked on the line that I set the interpreter on for iOS. A couple of you have already commented there, but others might not be aware.

I am using Entity Framework, and that is likely a main reason why I set it.

https://github.com/dotnet/runtime/issues/74015

[symbiogenesis]

Here is an example of something that seems to work for me on Debug mode, but fails on Release mode

In this branch there is a converter that is marked as internal. On Windows and/or in Debug mode it works, but on Android release mode it crashes with:

System.MethodAccessException: Method Maui.DataGrid.SortDataTypeConverter..ctor()' is inaccessible from method YourProject.Views.UsersPage.InitializeComponent()' at YourProject.Views.UsersPage.InitializeComponent...

To reproduce the error, just run the sample project in Android release mode. It should crash on launch.

Changing SortDataTypeConverter to public fixes it, but I don't see why Release mode should be special, so it seems like a bug.

To retrieve the exception in Release mode I have subscribed to AppDomain.CurrentDomain.UnhandledException and TaskScheduler.UnobservedTaskException from the MainActivity in the Android platform code, and logged to the console from there.

https://github.com/symbiogenesis/Maui.DataGrid/tree/android-release-mode

[rdavisau]

Is it possible in new dotnet to do the equivalent of the Xamarin.iOS --interpreter=-all (aot everything, but allow the interpreter to handle codegen)?

[rolfbjarne]

Is it possible in new dotnet to do the equivalent of the Xamarin.iOS --interpreter=-all (aot everything, but allow the interpreter to handle codegen)?

This should work the same:

<PropertyGroup>
    <MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>

[Streamtech-dev]

I was experiencing crashes when writing to sqlite in release mode on ios. Using the interpreter stopped that happening. It also made some other minor things work again like colour changes to controls. I only experienced these problems with ios, which worked fine in debug mode. Other platforms all worked OK in both debug and release. I used the example from above and changed Debug to Release to make it work.

<PropertyGroup>
	<UseInterpreter Condition="'$(UseInterpreter)' == '' and '$(Configuration)' == 'Release'">True</UseInterpreter>
	<MtouchLink Condition="'$(MtouchLink)' == '' and '$(Configuration)' == 'Release' and '$(UseInterpreter)' == 'true'">None</MtouchLink>
</PropertyGroup>

[mattleibow]

Using the interpreter by default may cause perf issues for everyone - even those who don't need it.

The opposite is that some portion of the users may have issues in release and waste time trying to fix it.

Both are bad, but which is the most significant number? Is there a way to have partial interpretation so that the libraries that have issues can use the interpreter but the rest of the app is AOT?

Also, have we been able to determine what causes the issues? Maybe there is some method use in EF for example that needs to have an AOT friendly implementation on some platforms?

TL;DR: I am suggesting that if there is an issue with a few libraries for a few people then maybe we can fix those instead of negatively impacting performance for all users.

[cdavidyoung]

"The opposite is that some portion of the users may have issues in release and waste time trying to fix it."

That is my case. My iOS app CacheAll worked perfectly in debug and so I published the release version on the app store. This also worked fine until you went to save and then it would silently crash with no crash report.

Fortunately, a little Googling revealed the <UseInterpreter>true</UseInterpreter> trick, which I put in the iOS release configuration of the .csproj file. I verified that this corrected the problem by first running in VS Mac the release version without this fix and noting that the app crashed when saving. Then I added this fix, did a clean all and deleted bin and obj, and ran again in VS Mac. Now the app does not crash.

Subsequently I uploaded to the app store and downloaded the app via TestFlight, which also showed that the crash was fixed.

BTW, so far I have not noticed a performance issue. @rolfbjarne Does UseInterpreter only affect the code that was crashing or does it affect the execution of the app overall?

[Rory-Reid]

I've just hit this myself after developing on mac for months but having no mac users, until today, precisely because it works in debug but not release.

While I understand the concerns for performance, it's also not a great out of the box experience for MAUI, and given that it's not an obvious tradeoff to consider (I don't remember seeing it in any documentation or tutorials I hit, anecdotally) it feels like it could be quite devastating to run into for people planning to use MAUI for mac and iOS if they depend on a library that needs it (for me it was Microsoft.Data.Sqlite I believe).

Tough call, but I would like this to be more obvious/easier to deal with than it is today. The Microsoft MAUI csproj templates have a bespoke commented-out-by-default section for tizen, in a similar vein, it might be nice to see something like this added there if not enabled by default:

<!-- Uncomment to enable the interpreter. This is needed in release if you or your dependencies use reflection -->
<!-- <PropertyGroup Condition="$(TargetFramework.Contains('-maccatalyst')) Or $(TargetFramework.Contains('-ios'))">-->
<!--     <UseInterpreter>true</UseInterpreter>-->
<!-- </PropertyGroup>-->

[mattleibow]

Yeah,you changed me :) I like this idea... We can have it enabled actually and then maybe have some publishing doc that focuses on optimization where we can talk linking, reflection and interpreters...

Should this be on from the workloads and you opt out... This will affect existing users. Or should this be a templates thing and only be something for new users.

Today we enable for release builds implicitly so maybe we just always set it for ios/maccatalyst and just release for Android. That is semi inconsistent but might be the best. The the project can override however they like but at least the default always works.

[mattleibow]

@davidbritch @rolfbjarne @jonathanpeppers do we have a doc on all the options that are good for optimizing maui apps? Obviously there are millions of buttons, but more the ones that most people use.

Should the interpreter be there along with the pros and cons?

[RobTF]

To the comment;

Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.

This sort of points the finger at customers, rightly so in many cases I'm sure - however there are a few nuances here;

• Rightly or wrongly, we as .NET developers have been trained to think that building in release mode is intrinsically safe and simply provides a performance increase. It is so rare I can't even think of a case where other frameworks such as WinForms, WPF, ASP.NET etc. ever have this issue of release mode being significantly less stable, especially in weird subtle ways. The significance and danger of this should be documented more clearly - more fairly seasoned Xamarin developers may understand this as it is the same sort of issue which applies to linking, but newcomers likely won't.
• "Testing our apps" is easy to say, but it might be a job that takes months for any non trivial app. We have enough of that when debugging our own code. I can imagine that even with the best of intentions edge cases may slip through.
• When the problem does occur - the error traces are garbage. What leads developers to turning the interpreter on is little more than shotgun debugging e.g. "I'll try switching on the crash-less flag". This is a tooling issue.
• This is not a third party library problem - most of the errors solved by this flag that I've seen stem from bindings.

Quick question: Does the interpreter exist in Xamarin? If not, with it switched on does this negate any of the performance enhancements of MAUI vs. Xamarin? I don't ever recall it being mentioned prior to MAUI.

[jonathanpeppers]

Rightly or wrongly, we as .NET developers have been trained to think that building in release mode is intrinsically safe

The trimmer does output the message:

Optimizing assemblies for size may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink

This is one source of when Release builds go wrong, but maybe this could be a warning instead of a message?

The interpreter is more of a setting for iOS, as it enables more System.Reflection.Emit-related APIs to work in a JIT-not-allowed environment. There really isn't a reason to turn the interpreter on for Android, as it is only used for hot reload scenarios.

[rolfbjarne]

To the comment;

Some customers don't even test their apps in Release mode, they go straight to trying to publish and they're obviously rather upset when their apps are rejected because they crash at launch.

This sort of points the finger at customers, rightly so in many cases I'm sure

I fully agree with all your comments, and I didn't mean to point any fingers at anyone but ourselves, and I'm sorry if it came off that way.

Quick question: Does the interpreter exist in Xamarin? If not, with it switched on does this negate any of the performance enhancements of MAUI vs. Xamarin? I don't ever recall it being mentioned prior to MAUI.

Yes, the interpreter has been around for a few years now, but it was an experimental feature in Xamarin (you had to opt in).

This meant few people tried it, and even fewer ran into differences between Debug and Release.

One reason this has become such an annoying problem is because the interpreter is required for Hot Reload (which is new in MAUI), so it was turned on by default for Debug, and thus a lot of people would happily (and rightfully so) code away, using features that would require (by accident or by design) the interpreter.

[rolfbjarne]

This is one source of when Release builds go wrong, but maybe this could be a warning instead of a message?

That's not very useful for iOS, because for internal reasons the linker is always enabled, and a warning you can't fix is a warning that will be ignored.

[RobTF]

One reason this has become such an annoying problem is because the interpreter is required for Hot Reload

Thanks @rolfbjarne,

Couple of observations;

If you mean hot reload for code (as opposed to XAML which I assume doesn't need it as it worked prior to the interpreter?) then I'd say I'd rather not have the code hot reload and get rid of the interpreter problem altogether, as the moment I diverge away from the "hello world" MAUI example, code-level hot-reload stops working anyway and none of my team use it. The XAML hot reload is much more important for iterating UIs, but as I say that seems to have been working in Xamarin long before the interpreter. Edit: Ah, I realise you might be hesitant to do this as MAUI pushes code-based (as opposed to XAML based) UI as a first class concept, is that correct?

Secondly, there seems to be a thing of third party libraries accidentally requiring the interpreter (such as EF Core), but if Xamarin didn't use the interpreter why did this never crop up? and how did libraries such as EF Core work ok back then? Is there something fundamentally different about how these libraries work now or is simply the presence of the interpreter enough to create the potential for such a requirement?

Also, I'm guessing my team never used the Xamarin interpreter (as we never knowingly opted in), but with MAUI we will need to use the interpreter for release builds, so are there any figures on how performance compares between non-interpreted Xamarin vs. interpreted MAUI? Just want to get a feel for whether we should expect to see performance degradation in any app ports whilst we migrate.

[gkarabin]

Two questions around crash report weirdness:

1. If the interpreter is enabled, is it obscuring all stack traces or just ones where a code path requires interpretation (e.g., some reflection emit call). I had assumed that parts of the app remain AOT compiled and are not interpreted, with normal stack traces. If it’s just some crash reports that are useless it’s not quite so bad.
2. If the app crashes when in the interpreter, does the interpreter itself have useful diagnostics that could be added to the crash dump to compensate? I realize that’s probably a whole new feature and ours of scope - just curious.

[RobTF]

@gkarabin That's a good point, additionally in some cases the issue is the opposite - I think the issue isn't often the interpreter itself, but lack of the interpreter when an app tries to execute code that requires it, which can't be known ahead of time.

You almost need two error scenarios and appropriate messages;

InterpreterNotEnabledException: Failed to execute abc due to xyz. This operation requires the MAUI interpreter to be enabled on this platform, see http://aka.ms/maui-interpreter for more information.

InterpreterFailedException: Failed to execute abc due to xyz. An error occurred within the MAUI interpreter.
Diagnostics:
-------------
<something here - last instruction, method name, etc.>

[rolfbjarne]

Secondly, there seems to be a thing of third party libraries accidentally requiring the interpreter (such as EF Core), but if Xamarin didn't use the interpreter why did this never crop up? and how did libraries such as EF Core work ok back then? Is there something fundamentally different about how these libraries work now or is simply the presence of the interpreter enough to create the potential for such a requirement?

Each of these cases are different, and would have to be investigated separately. I know that there were some regressions in some cases when merging old Mono-based code from Xamarin with Microsoft-based code (because the Mono-based code was implemented to be safe in a FullAOT scenario - i.e. no interpreter/jit - while Microsoft historically never had FullAOT support, so the code was written differently). That said, I don't know if EFCore is in that bucket. In either case, I would recommend filing issues in dotnet/runtime about these cases, and we'll investigate and try to figure out what can be done.

Also, I'm guessing my team never used the Xamarin interpreter (as we never knowingly opted in), but with MAUI we will need to use the interpreter for release builds, so are there any figures on how performance compares between non-interpreted Xamarin vs. interpreted MAUI? Just want to get a feel for whether we should expect to see performance degradation in any app ports whilst we migrate.

That's extremely dependent on the app. If your app is a game doing a lot of math in C#, it will be much slower (probably excruciatingly so). If you're app does very little CPU heavy code in C#, then the performance slowdown might even be unmeasurably small.

Also note that it's possible to disable the interpreter on an assembly basis, so if you find out that you have a particular assembly where you're doing a lot of math, you can disable the interpreter for that assembly.

1. If the interpreter is enabled, is it obscuring all stack traces or just ones where a code path requires interpretation (e.g., some reflection emit call). I had assumed that parts of the app remain AOT compiled and are not interpreted, with normal stack traces. If it’s just some crash reports that are useless it’s not quite so bad.

Any stack frame executed by the interpreter will be rather useless. The interpreter can be disabled on a per-assembly basis, so it's possible to have some frames from some assemblies accurately depicted in any crash reports, while others will just say the interpreter is doing stuff.

2. If the app crashes when in the interpreter, does the interpreter itself have useful diagnostics that could be added to the crash dump to compensate?

We can't add anything to the built-in crash report on iOS at runtime. The crash reporter is an out-of-process executable that examines the process right before it's terminated by the OS, and writes out the results (i.e. the crash report).

If the crash originates from an unhandled managed exception, then that exception might contain more information, and in fact it does sometimes. Example exception:

System.ExecutionEngineException: Attempting to JIT compile method ... while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.

There's a link there, but unfortunately it doesn't say anything about the interpreter. However, I've filed https://github.com/MicrosoftDocs/xamarin-docs/issues/3546 to try to rectify that.

[gkarabin]

Thanks for the awesomely thorough explanations, @rolfbjarne. The per-assembly approach has legs for my apps - the performance critical stuff is limited to native code (C++) and just a few assemblies.

I wonder if there's room in the iOS Release build process to look for assemblies that would require the interpreter to run correctly, and fail the build if the assembly doesn't use the interpreter? Dunno if it would just be heuristics or if you could be 100% sure.

That would give people a way to know about the problem before it's found in production. I realize its scope creep compared to the original point of this issue, but it might hit the sweet spot if it can be done without blowing up build times.

[rdavisau]

To my understanding, in most (all?) situations, none of the assemblies available at compile time should require the interpreter to execute. "Problematic" libraries emit new code into new dynamic assemblies that only exist at runtime. These would typically be JIT'd, but here need to be interpreted, and (probably/practically) can't be named in your csproj ahead of time.

So I think the solution in dotnet ios, is the same as in Xamarin.iOS - this:

<PropertyGroup>
    <MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>

and it might be a reasonable project default.

[baaaaif]

I became aware of this fix in the EF Core 8 RC1 blog post https://github.com/dotnet/efcore/issues/29725 and now I'm wondering whether this might be related to this topic? Does this make UseInterpreter obsolete, at least for using EF Core Sqlite for iOS?

(https://devblogs.microsoft.com/dotnet/announcing-ef8-rc1/)

[MichaelLHerman]

Just for the record, it's not just EF Core. I have also experienced problems that only appear in Release mode when using Autofac with its autogenerated Factory registrations

https://autofac.readthedocs.io/en/latest/resolve/relationships.html#dynamic-instantiation-func-b https://github.com/autofac/Autofac/issues/1398#issuecomment-1771008802

I'll be switching over to Microsoft DI

[dirivero]

Any news on this one? Wouldn't the setting below be a sensible default for iOS as @rdavisau and @rolfbjarne pointed out?

<PropertyGroup Condition="$(TargetFramework.Contains('-ios'))">
    <MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>

Another common scenario that fails is Linq Expressions with more that 2 parameters: https://github.com/dotnet/runtime/issues/94063

[DDHSchmidt]

Any news on this one? Wouldn't the setting below be a sensible default for iOS as @rdavisau and @rolfbjarne pointed out?

<PropertyGroup Condition="$(TargetFramework.Contains('-ios'))">
    <MtouchInterpreter>-all</MtouchInterpreter>
</PropertyGroup>

Another common scenario that fails is Linq Expressions with more that 2 parameters: dotnet/runtime#94063

Actually, after some more troubles with that parameter I'd advise against that as a default. Even with -all we still experienced crashes in our iOS-Release builds, the latest one in the Maui Community Toolkit :/

Finding the root culprit/assembly is a very manual & cumbersome process, even after @rolfbjarne gave me some tips on Discord to speed things up (Kudos for that 👍) We still put up with <MtouchInterpreter>-all</MtouchInterpreter> because our migrated Maui app is still slow as ass, compared to the old Xamarin version and we need to squeeze out every ounce of performance we can get 😕

NativeAOT is an even bigger pita. The crashes are even more numerous and the stacktraces/hints next to nonexistant. At the current pace I expect NativeAOT not to be usable for anything but console apps before .Net 11, but I digress...

I would recommend to stick with <UseInterpreter>true</UseInterpreter> as a default for Debug & Release builds unless you consider all MAUI users to be either compiler-experts or masochists.