[Bug]: Too many open files on Linux

[hykilpikonna]

Are you on the latest version of AssetRipper?

Yes, I'm on the latest alpha-build of AssetRipper.

Which release are you using?

Linux x64

Which game did this occur on?

No response

Which Unity version did this occur on?

5.6.4f

Is the game Mono or IL2Cpp?

Mono

Describe the issue.

System.IO.IOException: Too many open files (even when I set ulimit to an excessively large number)

Please close some files after opening.

Relevant log output

System.IO.IOException: Too many open files : '/tmp/package/[Redacted]/StreamingAssets/assets/[Redacted].ab'
   at Interop.ThrowExceptionForIoErrno(Interop.ErrorInfo, String, Boolean) + 0x19
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String, Interop.Sys.OpenFlags, Int32, Boolean, Boolean&, Func`4) + 0x13f
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String, FileMode, FileAccess, FileShare, FileOptions, Int64, UnixFileMode, Int64&, UnixFileMode&, Boolean, Boolean&, Func`4) + 0x11c
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String, FileMode, FileAccess, FileShare, FileOptions, Int64, Nullable`1) + 0x8e
   at System.IO.FileStream..ctor(String, FileMode, FileAccess, FileShare, Int32, FileOptions, Int64) + 0x9a
   at AssetRipper.IO.Files.Streams.MultiFile.MultiFileStream.OpenRead(String) + 0xb6
   at AssetRipper.IO.Files.SchemeReader.LoadFile(String) + 0x26
   at AssetRipper.Assets.Bundles.GameBundle.LoadFilesAndDependencies(IEnumerable`1, IDependencyProvider) + 0x9b
   at AssetRipper.Assets.Bundles.GameBundle.InitializeFromPaths(IEnumerable`1, AssetFactoryBase, IDependencyProvider, IResourceProvider, UnityVersion) + 0x36
   at AssetRipper.Assets.Bundles.GameBundle.FromPaths(IEnumerable`1, AssetFactoryBase, IDependencyProvider, IResourceProvider, UnityVersion) + 0x4f
   at AssetRipper.Import.Structure.GameStructure.InitializeGameCollection(UnityVersion) + 0x176
   at AssetRipper.Import.Structure.GameStructure..ctor(List`1, CoreConfiguration) + 0xfc
   at AssetRipper.Import.Structure.GameStructure.Load(IEnumerable`1, CoreConfiguration) + 0x3b
   at AssetRipper.Export.UnityProjects.ExportHandler.Load(IReadOnlyList`1) + 0x19a
   at AssetRipper.GUI.Web.GameFileLoader.LoadAndProcess(IReadOnlyList`1) + 0x46
   at AssetRipper.GUI.Web.Pages.Commands.LoadFolder.<AssetRipper-GUI-Web-Pages-ICommand-Execute>d__0.MoveNext() + 0x179
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at AssetRipper.GUI.Web.Pages.Commands.<HandleCommand>d__6`1.MoveNext() + 0xaf
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
   at AssetRipper.GUI.Web.ErrorHandlingMiddleware.<InvokeAsync>d__0.MoveNext() + 0x9f

[github-actions[bot]]

Thank you for submitting your first issue here. Please be sure you have uploaded your AssetRipper.log file. It is in the same folder as the exe file. :)

[hykilpikonna]

(started opening the directory)
18:45:09.3 Open FD: 23
18:45:09.8 Open FD: 23
18:45:10.3 Open FD: 283
18:45:10.8 Open FD: 448
18:45:11.3 Open FD: 508
18:45:11.8 Open FD: 556
18:45:12.3 Open FD: 568
18:45:12.8 Open FD: 133
18:45:13.3 Open FD: 207
18:45:13.8 Open FD: 269
18:45:14.3 Open FD: 311
18:45:14.8 Open FD: 101
18:45:15.3 Open FD: 149
18:45:15.8 Open FD: 205
18:45:16.3 Open FD: 84
18:45:16.8 Open FD: 138
18:45:17.3 Open FD: 202
18:45:17.8 Open FD: 161
18:45:18.3 Open FD: 161
18:45:18.8 Open FD: 249
18:45:19.3 Open FD: 273
18:45:19.8 Open FD: 98
18:45:20.3 Open FD: 128
18:45:20.8 Open FD: 117
18:45:21.3 Open FD: 161
18:45:21.8 Open FD: 235
18:45:22.3 Open FD: 763
18:45:22.8 Open FD: 57
18:45:23.3 Open FD: 442
18:45:23.8 Open FD: 861
18:45:24.3 Open FD: 1024
(crashed because the open file limit of 1024 is reached)
18:45:24.8 Open FD: 38
18:45:25.3 Open FD: 38
18:45:25.8 Open FD: 38
18:45:26.3 Open FD: 38

[ds5678]

f423c9ed7a53b25420abd88670ed53b49389eac9 might fix this.

[UlyssesZh]

f423c9e might fix this.

No. Still reproducible on 1.0.9.

[ds5678]

How do you propose I solve the problem then?

[UlyssesZh]

I have no clue.

I found that 0.3.4.0 doesn't have this problem. Maybe we can bisect to see which commit introduced the bug.

[ds5678]

I suspect 0.3.4.0 just discovered less files to rip, rather than being better in any way.