Header enumeration issues

[itssimple]

trafficstars

It works most of the time, but when we have many requests at the same time for the same resource, we sometimes end up with this (and other exceptions, that doesn't make any sense)

Version of SDK: 4.0.5 Using .NET 6.0

Exception:

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.List`1.Enumerator.MoveNext()
   at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)
   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)
   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()
   at Minio.ResponseResult.get_Headers() in /root/.q/sources/minio-dotnet/Minio/ResponseResult.cs:line 113
   at Minio.MinioClient.StatObjectAsync(StatObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 1095
   at Minio.MinioClient.getObjectHelper(GetObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/Helper/OperationsHelper.cs:line 50

Code (snippet) used to get the file:

var outFile = new MemoryStream();

var getObjectArgs = new GetObjectArgs()
.WithBucket(bucketName)
.WithObject(minioObjectName)
.WithCallbackStream((s) =>
	s.CopyTo(outFile)
);

await _minioClient.GetObjectAsync(getObjectArgs, cancellationToken: cancellationToken);

outFile.Seek(0, SeekOrigin.Begin);

return File(outFile, dbfr.MimeType, null, true);

[itssimple]

Exception with Object Reference-error

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)
   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)
   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()
   at Minio.ResponseResult.get_Headers() in /root/.q/sources/minio-dotnet/Minio/ResponseResult.cs:line 113
   at Minio.MinioClient.StatObjectAsync(StatObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 1095
   at Minio.MinioClient.getObjectHelper(GetObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/Helper/OperationsHelper.cs:line 50

Same request as above, just repeated a few times.

[itssimple]

And then the Index out of range

System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)
   at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue)
   at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext()
   at Minio.ResponseResult.get_Headers() in /root/.q/sources/minio-dotnet/Minio/ResponseResult.cs:line 113
   at Minio.MinioClient.StatObjectAsync(StatObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 1095
   at Minio.MinioClient.getObjectHelper(GetObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/Helper/OperationsHelper.cs:line 50

[ebozduman]

@itssimple

I cannot reproduce the issue.

I definitely need more info or the repro steps and please clarify/give more detail on how you "... have many requests at the same time for the same resource".

[itssimple]

I'll see if I can make a small repo solution.

[ebozduman]

One more thing: It is almost always suggested to add/call ConfigureAwait(false) after an awaited async method, like: await _minioClient.GetObjectAsync(getObjectArgs, cancellationToken: cancellationToken).ConfigureAwait(false);

It means that you do not care if the code after the await, runs on the captured context or not, unless otherwise is true in your scenario. This will also help performance.

Could you apply this change, if it is ok, and then try to reproduce the issue?

[itssimple]

After applying .ConfigureAwait(false) to all awaited calls, it is still happening.

Will try to reproduce this in a small reproduction repository during the day.

[itssimple]

Can't reproduce in a small repo (localhost), will try to deploy it internally and see if it reproduces

[itssimple]

@ebozduman Here's a repo that reproduces when deployed behind HAProxy. So my guess is that something in our HAProxy does something strange.

https://github.com/MultinetInteractiveOSS/MinioHeaderRepro

[ebozduman]

@itssimple

Thank you for your time and the effort to create the "repro repo".

So, if you suspect HAProxy could be the culprit, could you try a couple of things:

1. Run without the proxy, if possible, and see if the issue can still be reproduced,
2. Run with, let's say Nginx proxy or with Squid (Linux only) proxy to see if the problem is specific to HAProxy

Meanwhile, for me to try to repro here, could you copy/paste the HAProxy config file?

I started running the test (without Proxy). I assumed it can be run (not yet checked the code), correct? How long does it take, "ballpark", to repro the issue in your case?

[itssimple]

The repo needs a few environment variables to be set (to get a Minio instance working, with a specific file)

I'm currently home sick, so I don't have the HAProxy config available right now. But it works fine without the proxy (when I tested it locally), but as soon as I put it behind HAProxy it started misbehaving.

Will see if I can make a small HAProxy (+ version) config that's able to reproduce the error, when I'm back at work.

[ebozduman]

Sounds good. It looks like this is not a MinIO issue, but we'll wait for your input and then close the issue if we decide there is nothing we can do about it, including may be a better error messaging or a recommendation to fix or to workaround the issue, etc. Thank you

[itssimple]

Looks like I can replicate it, just by having it deployed on our Windows servers, through IIS.

Windows Server 2019 Standard (1809, Build 17763.3165) IIS 10

But when I'm debugging locally, it works fine.

~~Will try to deploy locally as well, to see if it replicates on Windows 10 Enterprise, IIS 10~~

Edit: It does not replicate on Windows 10 Enterprise, will try to deploy repro on a clean Windows 2019.

[itssimple]

It's probably something with our setup, because a clean setup with a W2019 works just fine..

[ebozduman]

ok @itssimple. Thank you for confirming it. The issue is closed.