UEGitPlugin causes a massive slowdown on file write

[sidheart]

Summary UEGitPlugin degrades file write performance, which is an incredibly common operation in Unreal Engine 5. Copying folders with a large amount of files, deleting a large number of files, or even creating large numbers of files in engine while retargeting as an example takes several minutes instead of seconds when UEGitPlugin is enabled.

Steps to Reproduce

1. Enable UEGitPlugin
2. Copy a folder with 100 untracked files in git from one location to another inside Unreal Engine 5 and note the time to completion
3. Disable UEGitPlugin and restart the engine
4. Copy the same folder to another location inside Unreal Engine 5 and note the time to completion
5. The time difference shows that UEGitPlugin causes degraded performance
6. Repeat steps 1-5 for git lfs tracked files and note the same degradation. You can also verify that the degradation occurs for deletion and creation.

Workaround and Next Steps

1. You can disable the UEGitPlugin when performing file operations. However, this shouldn't be a requirement. If you forget and have unsaved changes, you might have to wait for an hour for simple operations to finish depending on the size.
2. I will help you resolve this issue if you can point me to the logic that handles checking locks and invoking the plugin. My guess is that either the plugin is making a network hop for every single file or some other expensive operation is invoked PER FILE.

[mastercoms]

Do you know if this is slower than any other source control plugin? There are some inefficiencies baked into the engine as well.

[sidheart]

Do you know if this is slower than any other source control plugin? There are some inefficiencies baked into the engine as well.

I only use git, so I'm not sure. Is the inefficiency something that forces you to run plugin logic per file instead of as a single bulk operation? It seems like the correct logic would be on a bulk operation resolve all locks at once with a single network hop + dialogue window, then perform the intended action (e.g. copy, delete, move).

I'm also running the latest release, not master. I wonder if the most recent commit fixes this behavior for untracked files?

[mastercoms]

So to be clear, Git LFS does not support batch unlocking/locking APIs, nor even multi-threading of unlock/lock for multiple files. So we modified Git LFS to add multi-threading, and also only do a single authentication request for a series of locks/unlocks, when the original project redos auth for every single file. Unfortunately, there's not much further that can be done on that since that's all we can do for the single file protocol.

As far as what happens in the engine, it requests a lot of data updates which aren't necessary for Git, but are mostly workarounds for Perforce. When I last looked at this, this was where the majority of the time was going, not any lock/unlock operation in the plugin.

These were the fixes we did to the engine in UE4 to fix this, but we haven't ported these patches yet to UE5.

https://github.com/ProjectBorealisTeam/UnrealEngine/commit/28ab36e92a98755ea896108bd61ac6729ad42b02

https://github.com/ProjectBorealisTeam/UnrealEngine/commit/c117a115666adeb2d555aae668fa682478c242bf

[sidheart]

I hate Perforce with a passion and I hate that Epic gives them preferential treatment. I can't read those commits, I think they are probably privacy protected for the "ProjectBorealisTeam". When I click your links on my account it leads to a 404.

Anyway I'm very interested in working out a solution to this problem. My team uses this plugin for development and it's mostly exceptional. Let me know if you're able to make those commits visible. If not a few code pointers to the UE4/UE5 Perforce data requests would be helpful so I can think about how to rip them out.

[mastercoms]

https://github.com/ProjectBorealis/UE-Access

Have you set up GitHub code access to Unreal Engine from Epic Games?

[sidheart]

Got access, thanks. I'll take a look.

As far as what happens in the engine, it requests a lot of data updates which aren't necessary for Git, but are mostly workarounds for Perforce. When I last looked at this, this was where the majority of the time was going, not any lock/unlock operation in the plugin.

I'll try the solution and see if it works. If you remember the tools you used to profile where the engine was taking the most time, that would also be extremely helpful.

[mastercoms]

You can use any sampling profiler, really. I think I mainly used the Visual Studio profiler at the time.