docfx icon indicating copy to clipboard operation
docfx copied to clipboard

[Bug] System.IO.IOException: The process cannot access the file

Open VaclavElias opened this issue 2 years ago • 20 comments

Describe the bug

I run docfx en/docfx.json and get this error below, but it is very rare and random. Maybe worth to log it here. Maybe it happens once per 20-30 runs in my case.

System.IO.IOException: The process cannot access the file 'D:\Projects\GitHub\stride-docs-next\en\_site\api\Valve.VR.PropertyWrite_t.html' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at Microsoft.DocAsCode.Common.ManifestFileWriter.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/ManifestFileWriter.cs:line 65
   at Microsoft.DocAsCode.Plugins.RootedFileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Plugins/RootedFileAbstractLayer.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.HtmlPostProcessor.Process(Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs:line 44
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandler.Handle(List`1 postProcessors, Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/PostProcessorsHandler.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.DocumentBuilder.Build(IList`1 parameters, String outputDirectory) in /_/src/Microsoft.DocAsCode.Build.Engine/DocumentBuilder.cs:line 173
   at Microsoft.DocAsCode.SubCommands.DocumentBuilderWrapper.BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, String baseDirectory, String outputDirectory, String pluginDirectory, String templateDirectory) in /_/src/[Microsoft.DocAsCode.App/Helpers/DocumentBuilderWrapper.cs:line](http://microsoft.docascode.app/Helpers/DocumentBuilderWrapper.cs:line) 47
   at Microsoft.DocAsCode.RunBuild.Exec(BuildJsonConfig config, BuildOptions options, String configDirectory, String outputDirectory) in /_/src/[Microsoft.DocAsCode.App/RunBuild.cs:line](http://microsoft.docascode.app/RunBuild.cs:line) 31

To Reproduce

Just a regular run of docfx build.

Expected behavior

Probably it shouldn't happen. The processes shouldn't be locking the file?

Context (please complete the following information):

  • OS: Windows 11
  • Docfx version: 2.65.3

Another example:

image

VaclavElias avatar Apr 18 '23 14:04 VaclavElias

I have the same issue.

sakno avatar Apr 22 '23 14:04 sakno

Updated error 2.66.0

Error: System.IO.IOException: The process cannot access the file 'D:\Projects\GitHub\stride-docs-next\en\_site\manual\graphics\textures\index.html' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.File.Create(String path)
   at Microsoft.DocAsCode.Common.ManifestFileWriter.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/ManifestFileWriter.cs:line 52
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Create(RelativePath file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/FileAbstractLayer.cs:line 70
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Common/FileAbstractLayer/FileAbstractLayer.cs:line 136
   at Microsoft.DocAsCode.Plugins.RootedFileAbstractLayer.Create(String file) in /_/src/Microsoft.DocAsCode.Plugins/RootedFileAbstractLayer.cs:line 40
   at Microsoft.DocAsCode.Build.Engine.HtmlPostProcessor.Process(Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs:line 44
   at Microsoft.DocAsCode.Build.Engine.PostProcessorsHandler.Handle(List`1 postProcessors, Manifest manifest, String outputFolder) in /_/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/PostProcessorsHandler.cs:line 31
   at Microsoft.DocAsCode.Build.Engine.DocumentBuilder.Build(IList`1 parameters, String outputDirectory) in /_/src/Microsoft.DocAsCode.Build.Engine/DocumentBuilder.cs:line 161
   at Microsoft.DocAsCode.DocumentBuilderWrapper.BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, String baseDirectory, String outputDirectory, String pluginDirectory, String templateDirectory) in
/_/src/Microsoft.DocAsCode.App/Helpers/DocumentBuilderWrapper.cs:line 47
   at Microsoft.DocAsCode.RunBuild.Exec(BuildJsonConfig config, BuildOptions options, String configDirectory, String outputDirectory) in /_/src/Microsoft.DocAsCode.App/RunBuild.cs:line 30

VaclavElias avatar Apr 24 '23 20:04 VaclavElias

Same problem in 2.67.1 (2.67.1+9929fafef7443615f562bf47a49a3f783de3d375)

But not exactly rare for me, I would guess it happens once in every 4 tries.

TIS-Tim avatar May 30 '23 09:05 TIS-Tim

How many pages are you processing?

In my case it is rare with these:

Building 1620 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 303 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
Building 4 file(s) in TocDocumentProcessor(BuildTocDocument)...

VaclavElias avatar May 30 '23 10:05 VaclavElias

My project is way smaller I guess:

Building 2 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
Building 3 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
Building 3 file(s) in TocDocumentProcessor(BuildTocDocument)...
Building 72 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...

TIS-Tim avatar May 30 '23 10:05 TIS-Tim

That is interesting. I hope our input will help to find and resolve the issue.

VaclavElias avatar May 30 '23 10:05 VaclavElias

@TIS-Tim Is it convenient to share your project for me to repro?

yufeih avatar May 31 '23 03:05 yufeih

@yufeih

Once I tried to package the relevant files, I wanted to try to repro one more time, and it never happened.

Returning to my original project, with the solution opened in visual studio and continuing to work on that immediately triggered the issue again.

So my guess is it is probably not caused by the data, but by the workflow... like browsing the results, continue editing the sources during the build, having Visual Studio open on the project, something like that.

TIS-Tim avatar May 31 '23 07:05 TIS-Tim

I've tried docfx build command with reported docs content with related metadata.

But I can't reproduce IOException in my environment. (executed about 100 times) If there are specific procedure or conditions to reproduce problem. I would like to know.

Below is a description of the investigation


Execution logs

[01] Building 1620 file(s) in ResourceDocumentProcessor(ValidateResourceMetadata)...
[08] Building 5 file(s) in TocDocumentProcessor(BuildTocDocument)...
[07] Building 303 file(s) in ConceptualDocumentProcessor(BuildConceptualDocument=>CountWord=>ValidateConceptualDocumentMetadata)...
[12] Building 2707 file(s) in ManagedReferenceDocumentProcessor(BuildManagedReferenceDocument=>SplitClassPageToMemberLevel=>ValidateManagedReferenceDocumentMetadata=>ApplyOverwriteDocumentForMref=>FillReferenceInformation)...

Note: Leading number is thread id, It's appended to checking executed thread. for debug purpose.


Details

  1. IOException is occurred at HtmlPostProcessor (by reported logs)
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs#L85

  2. File::OpenRead operation is succeeded that is executed before HtmlPostProcessors handlers execution.
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/PostProcessors/HtmlPostProcessor.cs#L73

  3. HtmlPostProcessor's handlers(ValidateBookmark/RemoveDebugInfo) does't touch html file. So It's unlikely, this problem is caused by HtmlPostProcessor's processing. but other place or external process lock HTML files.

  4. PostProcessors are executed sequential order and HtmlPostProcessor is executed first.
    So It's unlikely this problem is caused by other PostProcessor's parallel operation.

  5. HTML file is created first at TemplateModelTransformer
    And if intentionally set exclusive lock to this file (by external process). IOException occurred here.
    https://github.com/dotnet/docfx/blob/db8ab17c5770fb008fe13e600340caed20db5cab/src/Microsoft.DocAsCode.Build.Engine/TemplateProcessors/TemplateModelTransformer.cs#L277

File lock related activity logs

I've get file access logs using Sysinternals's ProcessMonitor tool.
In my environment. Microsoft Defender(MsMpEng.exe) is working during docfx.exe execution.

image

filzrev avatar Jun 01 '23 10:06 filzrev

I'm also getting this error during a GitHub actions run of DocFX. Sometimes it works fine, sometimes it fails: https://github.com/Syncaidius/MoltenEngine/actions/runs/5689587121/job/15421370149

Exception:

[23-07-28 07:59:12.451]Error:System.IO.IOException: The process cannot access the file 'D:\a\MoltenEngine\MoltenEngine\docs\images/logo.svg' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at Microsoft.DocAsCode.Common.RealFileWriter.Copy(PathMapping sourceFileName, RelativePath destFileName)
   at Microsoft.DocAsCode.Common.FileAbstractLayer.Copy(RelativePath sourceFileName, RelativePath destFileName)
   at Microsoft.DocAsCode.Common.ManifestFileHelper.<>c__DisplayClass8_0.<Dereference>b__4(OutputFileInfo ofi)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass42_0`2.<PartitionerForEachWorker>b__1()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass176_0.<ExecuteSelfReplicating>b__0(Object <p0>)
	514 Warning(s)
	1 Error(s) 

Syncaidius avatar Jul 28 '23 16:07 Syncaidius

Interestingly, changing the docfx.json from:

"files": [
            "images/**"
          ]

to:

"files": [
            "images\\logo.svg"
          ]

Fixed it for me.

Syncaidius avatar Jul 28 '23 16:07 Syncaidius

I'm also encountered this problems with very rare rate. (I've set Windows Defender folder exclusion settings to docfx project folder)

And it seems also occurred at CI environment (Windows/ 2-CPU) https://github.com/dotnet/docfx/actions/runs/6404063982/job/17383853477

filzrev avatar Oct 04 '23 10:10 filzrev

A mysterious 👻🐛🤣. In the last 7 days, using DocFX daily, this issue occurred 3 times in a row on random files and then it was gone again.

Also, luckily this issue didn't occur through GitHub Action deployment to Azure App Service and in another app to GitHub Pages.

VaclavElias avatar Oct 04 '23 11:10 VaclavElias

Seems to be more obvious after perf improvements to #9314 . CI failed many times in a row.

yufeih avatar Oct 21 '23 01:10 yufeih

Is this problem still occurred on latest docfx? I've not encountered this problem about a month on local environment (and docfx CI build).

If it still reproduceable on latest docfx version. It might be better to add logics to logging what process cause file lock (For Windows). Similar to https://github.com/dotnet/msbuild/pull/9488.

filzrev avatar Dec 18 '23 17:12 filzrev

exclude folders you are writing to from windows security/defender:

https://support.microsoft.com/en-us/windows/add-an-exclusion-to-windows-security-811816c0-4dfd-af4a-47e4-c301afe13b26#:~:text=Go%20to%20Start%20%3E%20Settings%20%3E%20Update,%2C%20file%20types%2C%20or%20process.

gdimauro avatar Aug 28 '24 13:08 gdimauro

Same issue here. Excluding folders like @gdimauro suggest does not help, at least not for me. Still get errors like below randomly (not every time).

IOException: The process cannot access the file 'C:\Users\pcl\source\repos\cim\cim-serviceapp\docfx\_site\api\dotnet\CIM.ServiceApp.html' because it is being used by another process.
  at SafeFileHandle CreateFile(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
  at SafeFileHandle Open(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
  at ctor(string path, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)

However, setting the maxParallelism command line option to 1 like shown below seems to work.

dotnet tool run docfx build docfx.json --maxParallelism 1

perclausen avatar Oct 22 '24 15:10 perclausen

@perclausen Is it able to to provide complete stack trace information? If maxParallelism:1 settings resolve issue. It's expected to be occurred on outside of HtmlPostProcessor. (that reported on original issue)

filzrev avatar Oct 23 '24 12:10 filzrev

@perclausen Is it able to to provide complete stack trace information? If maxParallelism:1 settings resolve issue. It's expected to be occurred on outside of HtmlPostProcessor. (that reported on original issue)

Sure, here it is @filzrev:

 IOException: The process cannot access the file
  'C:\Users\pcl\source\repos\cim\cim-serviceapp\docfx\_site\docs\logging.html'
  because it is being used by another process.
    at SafeFileHandle CreateFile(string fullPath, FileMode mode, FileAccess access
       , FileShare share, FileOptions options)
    at SafeFileHandle Open(string fullPath, FileMode mode, FileAccess access,
       FileShare share, FileOptions options, long preallocationSize, UnixFileMode?
       unixCreateMode)
    at ctor(string path, FileMode mode, FileAccess access, FileShare share,
       FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
    at FileStreamStrategy ChooseStrategyCore(string path, FileMode mode,
       FileAccess access, FileShare share, FileOptions options, long
       preallocationSize, UnixFileMode? unixCreateMode)
    at FileStream Create(string path)
    at Stream Create(RelativePath file) in ManifestFileWriter.cs:48
    at Stream Create(RelativePath file) in FileAbstractLayer.cs:46
    at Stream Create(string file) in FileAbstractLayer.cs:82
    at Stream Create(string file) in RootedFileAbstractLayer.cs:34
    at Manifest Process(Manifest manifest, string outputFolder) in
       HtmlPostProcessor.cs:80
    at void Process(Manifest manifest, string outputFolder) in
       PostProcessorsManager.cs:41
    at void Build(IList<DocumentBuildParameters> parameters, string
       outputDirectory) in DocumentBuilder.cs:148
    at void BuildDocument(BuildJsonConfig config, BuildOptions options,
       TemplateManager templateManager, string baseDirectory, string
       outputDirectory, string templateDirectory) in DocumentBuilderWrapper.cs:42
    at string Exec(BuildJsonConfig config, BuildOptions options, string
       configDirectory, string outputDirectory) in RunBuild.cs:39
    at void <Execute>b__0() in BuildCommand.cs:25
    at int Run(LogOptions options, Action run) in CommandHelper.cs:48
    at int Execute(CommandContext context, BuildCommandOptions settings) in
       BuildCommand.cs:15
    at Task<int> Execute(CommandContext context, CommandSettings settings) in
       CommandOfT.cs:40
    at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext
       context, ITypeResolver resolver, IConfiguration configuration) in
       CommandExecutor.cs:166

perclausen avatar Oct 23 '24 13:10 perclausen

@perclausen Thanks for your response.

It seems exception is thrown from HtmlPostProcessor. This processor is expected to be executed on sequentially. And not affected by maxParallelism settings.

If it can provide files that can reproduce reported problems. I'll Investigate what's cause IOException,

filzrev avatar Oct 23 '24 13:10 filzrev

@filzrev Same here on latest docfx 2.78.2+3ff9ae7a9700029e0023a7ece18a723d351f1e9b, crashes every single time when using "memberLayout": "separatePages".

Every time it crashes is on a separate file as expected from a true concurrency issue. I can also confirm that the fix suggested by @perclausen works: dotnet docfx --maxParallelism 1.

Reverting back to the default layout or adding "outputFormat": "apiPage" to docfx.json both avoid the problem as well.

Stack trace below:

AggregateException: One or more errors occurred. (The process cannot access the file 
'\docs\_site\api\Bonsai.Shaders.Rendering.UpdateTransform.Process.html' because it is being used by another process.)
     IOException: The process cannot access the file '\docs\_site\api\Bonsai.Shaders.Rendering.UpdateTransform.Process.html'   
     because it is being used by another process.
       at SafeFileHandle CreateFile(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
       at SafeFileHandle Open(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode?
          unixCreateMode)
       at ctor(string path, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
       at FileStream Create(string path)
       at Stream Create(RelativePath file) in RealFileWriter.cs:27
       at Stream Create(RelativePath file) in FileAbstractLayer.cs:46
       at Stream Create(string file) in FileAbstractLayer.cs:82
       at Stream Create(string file) in RootedFileAbstractLayer.cs:34
       at void TransformDocument(string result, string extension, IDocumentBuildContext context, string destFilePath, ManifestItem manifestItem, out List<XRefDetails> 
          unresolvedXRefs) in TemplateModelTransformer.cs:269
       at ManifestItem Transform(InternalManifestItem item) in TemplateModelTransformer.cs:152
       at void <ProcessCore>b__0(InternalManifestItem item) in TemplateProcessor.cs:190
       at void <RunAll>b__0(TElement s) in DocumentExceptionExtensions.cs:95
       at void <ForWorker>b__1(ref RangeWorker currentWorker, long timeout, out bool replicationDelegateYieldedBeforeCompletion)
       at void <ForWorker>b__1(ref RangeWorker currentWorker, long timeout, out bool replicationDelegateYieldedBeforeCompletion)
       at void Execute()
  at void Run<TState>(ReplicatableUserAction<TState> action, ParallelOptions options, bool stopOnFirstFailure)
  at ParallelLoopResult ForWorker<TLocal,TInt>(TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action<TInt> body, Action<TInt,
     ParallelLoopState> bodyWithState, Func<TInt, ParallelLoopState, TLocal, TLocal> bodyWithLocal, Func<TLocal> localInit, Action<TLocal> localFinally)
  at ParallelLoopResult ForWorker<TLocal,TInt>(TInt fromInclusive, TInt toExclusive, ParallelOptions parallelOptions, Action<TInt> body, Action<TInt,
     ParallelLoopState> bodyWithState, Func<TInt, ParallelLoopState, TLocal, TLocal> bodyWithLocal, Func<TLocal> localInit, Action<TLocal> localFinally)
  at ParallelLoopResult ForEachWorker<TSource,TLocal>(IEnumerable<TSource> source, ParallelOptions parallelOptions, Action<TSource> body, Action<TSource,
     ParallelLoopState> bodyWithState, Action<TSource, ParallelLoopState, long> bodyWithStateAndIndex, Func<TSource, ParallelLoopState, TLocal, TLocal>
     bodyWithStateAndLocal, Func<TSource, ParallelLoopState, long, TLocal, TLocal> bodyWithEverything, Func<TLocal> localInit, Action<TLocal> localFinally)
  at ParallelLoopResult ForEach<TSource>(IEnumerable<TSource> source, ParallelOptions parallelOptions, Action<TSource> body)
  at void RunAll<TElement>(IEnumerable<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in
     DocumentExceptionExtensions.cs:84
  at void RunAll<TElement>(IReadOnlyList<TElement> elements, Action<TElement> action, int parallelism, CancellationToken cancellationToken) in
     DocumentExceptionExtensions.cs:68
  at List<ManifestItem> ProcessCore(List<InternalManifestItem> items, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:185 
  at List<ManifestItem> Process(List<InternalManifestItem> manifest, ApplyTemplateSettings settings, IDictionary<string, object> globals) in TemplateProcessor.cs:82   
  at List<ManifestItem> ProcessTemplate() in ManifestProcessor.cs:186
  at void Process() in ManifestProcessor.cs:53
  at void Handle(List<HostService> hostServices, int maxParallelism) in LinkPhaseHandler.cs:32
  at Manifest Build(DocumentBuildParameters parameters, IMarkdownService markdownService, CancellationToken cancellationToken) in SingleDocumentBuilder.cs:71
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs:124
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory, string
     templateDirectory, CancellationToken cancellationToken) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory, CancellationToken cancellationToken) in RunBuild.cs:39  
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options) in DefaultCommand.cs:31
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in CommandExecutor.cs:  
     166

glopesdev avatar Dec 20 '24 13:12 glopesdev

On further investigation, these warnings that show up for successful builds might hold a clue:

warning: DuplicateOutputFiles: Multiple input files would generate to the same output path overwriting each other. Please rename at least 1 of following input files to ensure that there will be only one file to generate to the output path: "api/Bonsai.Windows.Input.SetCursorPosition.yml, api/Bonsai.Windows.Input.SetCursorPosition.Process.yml"

I get about ~2,000 such warnings, and I'm assuming the IO locking error is because indeed some of the writers are parallelized writing to the same file, no idea why this would happen only for this particular configuration.

glopesdev avatar Dec 20 '24 13:12 glopesdev

@glopesdev Reported error seems to be caused on Build/LinkPhaseHandler phase. (Original reported issue is happens on PostProcessor phase (that is executed sequentially))

Build/LinkPhaseHandler is executed in parallel. So it is quite possible that an IOException may occur if warning: DuplicateOutputFiles warnings are recorded.

This is because the process is executed in following order.

  1. LinkPhaseHandler access same output file in parallel. (And cause IOException)
  2. Then duplicated files are reported as warning and removed at ManifestUtility.RemoveDuplicateOutputFiles.

It might need extra logics to detect duplicate file on step.1.

I get about ~2,000 such warnings It it able to provide minimal samples that cause these warnings. (When using "memberLayout": "separatePages")

filzrev avatar Dec 21 '24 05:12 filzrev

I've also encountered IOException on Build/LinkPhaseHandler phase. (With DuplicateOutputFiles warnings)

On my case. There are remaining "outputFormat": "markdown" output files on output folder. And these files cause conflict when using "outputFormat": "apiPage" settings.

Errors are resolved. after cleanup api temporary directory.

filzrev avatar Dec 31 '24 04:12 filzrev

Errors are resolved. after cleanup api temporary directory.

@filzrev thank you so much for looking into this, I can also confirm that this resolves the problem in my case!

Is there a command on docfx to cleanup the cache? or alternatively if this is a problem, should docfx cleanup the cache when changing output format?

glopesdev avatar Dec 31 '24 08:12 glopesdev

I am hitting now this error in 50-80% when I run it on my local PC.

Repository: https://github.com/stride3d/stride-community-toolkit Docs: https://github.com/stride3d/stride-community-toolkit/tree/main/docs Run: run.bat (which cleans _site) or docfx docfx.json --serve

VaclavElias avatar Jan 23 '25 23:01 VaclavElias

This bug is quite severe[^1] :-( I tried various workarounds from this ticket but it seems to happen very often and the workarounds do not seem to help too much.

edit: My PowerShell command to generate documentation is rm -Force -Recurse "api","_site"; docfx .\docfx.json --maxParallelism 1 --serve

edit: My exception

  at SafeFileHandle CreateFile(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
  at SafeFileHandle Open(string fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode?
     unixCreateMode)
  at ctor(string path, FileMode mode, FileAccess access, FileShare share, FileOptions options, long preallocationSize, UnixFileMode? unixCreateMode)
  at FileStream Create(string path)
  at Stream Create(RelativePath file) in ManifestFileWriter.cs:48
  at Stream Create(string file) in RootedFileAbstractLayer.cs:25
  at Manifest Process(Manifest manifest, string outputFolder, CancellationToken cancellationToken) in HtmlPostProcessor.cs:84
  at void Process(Manifest manifest, string outputFolder, CancellationToken cancellationToken) in PostProcessorsManager.cs:41
  at void Build(IList<DocumentBuildParameters> parameters, string outputDirectory, CancellationToken cancellationToken) in DocumentBuilder.cs:149
  at void BuildDocument(BuildJsonConfig config, BuildOptions options, TemplateManager templateManager, string baseDirectory, string outputDirectory,
     string templateDirectory, CancellationToken cancellationToken) in DocumentBuilderWrapper.cs:42
  at string Exec(BuildJsonConfig config, BuildOptions options, string configDirectory, string outputDirectory, CancellationToken cancellationToken) in
     RunBuild.cs:39
  at void <Execute>b__0() in DefaultCommand.cs:51
  at int Run(LogOptions options, Action run) in CommandHelper.cs:48
  at int Execute(CommandContext context, Options options, CancellationToken cancellationToken) in DefaultCommand.cs:31
  at int Execute(CommandContext context, TSettings settings) in CancellableCommandBase.cs:24
  at Task<int> Execute(CommandContext context, CommandSettings settings) in CommandOfT.cs:40
  at async Task<int> Execute(CommandTree leaf, CommandTree tree, CommandContext context, ITypeResolver resolver, IConfiguration configuration) in
     CommandExecutor.cs:166

[^1]: Using DocFX 2.78.3

MartyIX avatar Apr 17 '25 07:04 MartyIX

@MartyIX how random is your issue?

VaclavElias avatar Apr 17 '25 07:04 VaclavElias

It happens very very often right now. Like 3/4 attempts.

MartyIX avatar Apr 17 '25 07:04 MartyIX

Thanks.

I will do my tests with https://github.com/stride3d/stride-community-toolkit on clean PC install. If I confirm the issue, docfx devs can use my repo for testing and figuring out the issue.

The interesting part is that when my docs is generated through GitHub Actions, it never fails.

VaclavElias avatar Apr 17 '25 09:04 VaclavElias