RepositoryStatus sometimes crashes with an AccessViolationException

[jcansdale]

We've noticed that the GitHub Extension for Visual Studio has been crashing with an AccessViolationException. This often happens when a user previews a PR and the extension calls RetrieveStatus. This exception occurs randomly and is thrown by the git_status_list_new method.

This exception is most easily reproduced using 0.24 or 0.25.0-preview-0033, but it does also happen with 0.23.1 (but I haven't seen it using the repro below).

How to reproduce

1. Clone https://github.com/github/VisualStudio.git to C:\source\github.com\github\VisualStudio\.
2. Compile and run the following code in Release configuration using LibGit2Sharp v0.24:

        static void Main(string[] args)
        {
            var path = @"C:\source\github.com\github\VisualStudio";
            for (int count = 0; count < 1000; count++)
            {
                var status = new Repository(path).RetrieveStatus();
                Console.WriteLine(count + ": " + status.IsDirty);
            }
        }

I tried to create a standalone app, but wasn't able to trigger it with a very simple repo. I'm using the github\VisualStudio one because I'm most familiar it, but it does this with many others as well.

What to expect

The most common exception is as follows:

System.AccessViolationException was unhandled
  HResult=-2147467261
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=LibGit2Sharp
  StackTrace:
       at LibGit2Sharp.Core.NativeMethods.git_status_list_new(git_status_list*& git_status_list, git_repository* repo, GitStatusOptions options)
       at LibGit2Sharp.Core.Proxy.git_status_list_new(RepositoryHandle repo, GitStatusOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\Core\Proxy.cs:line 2931
       at LibGit2Sharp.RepositoryStatus..ctor(Repository repo, StatusOptions options) in C:\projects\libgit2sharp\LibGit2Sharp\RepositoryStatus.cs:line 60
       at LibGit2Sharp.RepositoryExtensions.RetrieveStatus(IRepository repository) in C:\projects\libgit2sharp\LibGit2Sharp\RepositoryExtensions.cs:line 696
       at Libgit2Repro.Program.Main(String[] args) in c:\users\passp\onedrive\documents\visual studio 2015\Projects\Libgit2Repro\Libgit2Repro\Program.cs:line 20
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()

But sometimes NotFoundExceptions are thrown as well.

Related issues / PRs

• Find out why / when LibGit2Sharp is crashing: github/VisualStudio#1315
• GitHub extension crashes when viewing pull requests: github/VisualStudio#1306
• "could not find [file] to open" exception thrown when opening a PR: github/VisualStudio#1326
• Move from LibGit2Sharp v0.24.0 to v0.23.1: github/VisualStudio#1316
• Add using to avoid crash in Release mode: github/VisualStudio#1311

[bording]

@jcansdale How many times do you have to run the program to trigger the crash? What framework are you using when you experience the crash?

Using 0.25.0-preview-0033, I've run it several times with both net461 and netcoreapp2.0, and I haven't seen it crash yet.

However, I wasn't able to fully clone https://github.com/github/VisualStudio.git because it has a private submodule that I don't have access to:

fatal: clone of '[email protected]:github/VisualStudioBuildScripts' into submodule path 'C:/source/github.com/github/VisualStudio/script' failed

That might be why I'm not able to see it crash. Do you know of another repository that exhibits the same problem that I can fully clone?

[grokys]

@bording ah yes, sorry that submodule is private - it contains our private keys etc. The build instructions cover it, you need to do a git submodule deinit script.

[grokys]

Regarding reproing the crash, are you running in Release mode? The crash doesn't happen in Debug.

[carlosmn]

It'd be interesting to see what the C stack trace looks like, but I certainly suspect the repository is being disposed and thus freed while we're in the middle of trying to run the status. It might need a sizeable repository in order for it to take long enough for GC to take effect.

[jcansdale]

@bording,

How many times do you have to run the program to trigger the crash? What framework are you using when you experience the crash?

Often I would run it once and it would crash within 10 iterations. Just now I struggled a bit to get the crash. It seems to crash pretty soon or not at all. Maybe try restarting the app a few times?

It did finally crash:

I've got a crash .dmp file from this if that would be useful?

[bording]

I'm definitely running in release mode.

I went ahead and re-cloned the repo and removed the script submodule, butI have not been able to see a single crash. I've run the app many times now, and it's working fine.

@grokys @jcansdale I'm assuming you've got the script module actually checked out when you're running into the problem? Have you tried it with a clone of the repo without the script submodule?

If the presence of script submodule isn't relevant to causing the crash, then there must be some other variable that isn't yet accounted for.

What OS are you running on? What version of the .NET Framework do you have installed, and which one are you targeting? Are running the app as 32 or 64 bit?

I'm not sure any of that would make a difference, but I'm trying to think of what else it could be.

@jcansdale The dmp file might be helpful, if you've got a way to share it.

[carlosmn]

Here's a stack trace of one of the times it crashed. The top points to https://github.com/libgit2/libgit2/blob/a5cf255b471ad7113247d552d5695db0cb720882/src/attrcache.c#L182 which is a call to the inlined https://github.com/libgit2/libgit2/blob/a5cf255b471ad7113247d552d5695db0cb720882/src/attrcache.c#L27-L36 which just looks up the entry in a dictionary.

It's still trying to look up address 0x14 which keeps looking like we cleaned up the repo, or at least the cache midway and we're trying to access the struct at NULL.

The crash is reproducing by opening the Avalonia PR page in GHfVS and clicking around a few different ones.

>    ntdll.dll!77e0b2e3()    Unknown
     [Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]    
     git2-a5cf255.dll!attr_cache_lookup(git_attr_file * * out_file, git_attr_file_entry * * out_entry, git_repository * repo, git_attr_session * attr_session, git_attr_file_source source, const char * base, const char * filename) Line 182    C
     git2-a5cf255.dll!git_attr_cache__get(git_attr_file * * out, git_repository * repo, git_attr_session * attr_session, git_attr_file_source source, const char * base, const char * filename, int(__stdcall*)(git_repository *, git_attr_file *, const char *) parser) Line 215    C
     git2-a5cf255.dll!git_ignore__push_dir(git_ignores * ign, const char * dir) Line 352    C
     git2-a5cf255.dll!filesystem_iterator_frame_push_ignores(filesystem_iterator * iter, filesystem_iterator_entry * frame_entry, filesystem_iterator_frame * new_frame) Line 1170    C
     git2-a5cf255.dll!filesystem_iterator_frame_push(filesystem_iterator * iter, filesystem_iterator_entry * frame_entry) Line 1353    C
     git2-a5cf255.dll!filesystem_iterator_advance_into(const git_index_entry * * out, git_iterator * i) Line 1576    C
     git2-a5cf255.dll!handle_unmatched_new_item(git_diff_generated * diff, diff_in_progress * info) Line 1055    C
     git2-a5cf255.dll!git_diff__from_iterators(git_diff * * out, git_repository * repo, git_iterator * old_iter, git_iterator * new_iter, const git_diff_options * opts) Line 1235    C
     git2-a5cf255.dll!git_diff_index_to_workdir(git_diff * * out, git_repository * repo, git_index * index, const git_diff_options * opts) Line 1377    C
     git2-a5cf255.dll!git_status_list_new(git_status_list * * out, git_repository * repo, const git_status_options * opts) Line 346    C

[hybridherbst]

I've been following the trail for all those libgit2sharp crashes I'm experiencing (I'm trying to use it from inside Unity, with random crashes all the time, most notably on domain reload but also in random situtations), which led me here.

Has there ever been a consensus on what causes this? Is the workaround (two years later) to still use 0.23? The same is still happening on 0.26.1 with libgit2 2.0.298, and also latest preview versions.

[jcansdale]

Iirc, it was caused by objects being collected before native methods return. This was counter intuative as a developer used to managed code. The fix was to always use using blocks for objects that are disposable (most libgit2sharp types).

I hope that helps. It was a while ago we last faced this issue.

[hybridherbst]

I was able to pinpoint a couple more cases where that happened and have something that feels "kinda stable" right now. Same as you, I went to "using() everywhere" (I was caching the repository object before).

Additionally, the bug that took me the longest to find was that inside the using(), I was using LINQ queries and forgot about deferred execution - when execution was actually happening at some random other point in time, the repository object was long gone and I got those native crashes. Forcing execution inside the using() block with ToList() solved that part of the issue.

[jcansdale]

I was caching the repository object before

Yup, I think we were caching repository objects as well. It gets particularly confusing when Debug builds work, but Release builds don't!

[hybridherbst]

And stepping through with debugger also always works - seems to prevent the repository object GC as well...

[DaveSenn]

I've the same problem with 0.26 and 0.27 preview:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at LibGit2Sharp.Core.NativeMethods.git_status_list_new(LibGit2Sharp.Core.git_status_list* ByRef, LibGit2Sharp.Core.git_repository*, LibGit2Sharp.Core.GitStatusOptions)
--------------------------------
   at LibGit2Sharp.Core.Proxy.git_status_list_new(LibGit2Sharp.Core.Handles.RepositoryHandle, LibGit2Sharp.Core.GitStatusOptions)
   at LibGit2Sharp.RepositoryStatus..ctor(LibGit2Sharp.Repository, LibGit2Sharp.StatusOptions)
   at LibGit2Sharp.Repository.RetrieveStatus(LibGit2Sharp.StatusOptions

Just seen https://github.com/libgit2/libgit2sharp/issues/1886 ... my issue also started after an update of VS2019 to 16.10.1. (This update also contained the latest .NET 5 SDK 5.0.7)