Can we attach the original exception to MinioException?

[haiduong87]

I randomly encounter this exception

Minio.Exceptions.ConnectionException: MinIO API responded with message=Connection error:One or more errors occurred. (The response ended prematurely.). Status code=0, response=One or more errors occurred. (The response ended prematurely.), content= at Minio.MinioClient.ParseErrorNoContent(ResponseResult response)

And there are more than one reasons for "The response ended prematurely."

• https://stackoverflow.com/questions/59590012/net-httpclient-accept-partial-response-when-response-header-has-an-incorrect
• https://github.com/dotnet/runtime/issues/47612

I think it's better if we know the original exception.

Propose changes:

• MinioException: public MinioException(string message, ResponseResult serverResponse) : base(GetMessage(message, serverResponse), serverResponse?.Exception) { ServerMessage = message; ServerResponse = serverResponse; }
• ResponseResult: public Exception Exception { get; }

[ebozduman]

@haiduong87

Do you remember, by any chance, what test(s) or specific api(s) were you running when you hit this issue?

[haiduong87]

@ebozduman

I use this GetObjectAsync public Task GetObjectAsync(string bucketName, string objectName, long offset, long length, Action<Stream> cb, ServerSideEncryption sse = null, CancellationToken cancellationToken = default) https://github.com/minio/minio-dotnet/blob/master/Minio/ApiEndpoints/ObjectOperations.cs#L77

It random happens. So I hope to get more information from the inner exception

[haiduong87]

@ebozduman

I have more information. I make my own version, and get this inner exception

System.AggregateException: One or more errors occurred. (The response ended prematurely.) ---> System.IO.IOException: The response ended prematurely. at System.Net.Http.HttpConnection.FillAsync() at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken) --- End of inner exception stack trace ---

[ebozduman]

@haiduong87 ,

Thank you for the information. I was having hard time trying to find out the reproduction steps to look into the issue.

I'll check the new info and see if I can make some progress.

Could you send me other test code you use to reproduce this problem (if there is a different scenario other than the one you've explained above) and tell me about your setup environment, like how you start your MinIO server, etc.?

Do you see any error message(s) on MinIO console, by any chance?

[haiduong87]

@ebozduman

I can only access to the client. Here's our structure:

• Minio server: on k8s (I don't have right on here, and I don't know if there's a log or not)
• Minio client:
  • An ASP.net (core 3.1) api server
  • store meta data in mongodb
  • receive get request, process (check authen, check meta data...) and make request to minio
  • we stream directly from minio to client (no buffer or temp file)

When the error occurs: it's a long time request (we log when having exception, and the error request is about 5 or more minutes before encounter the exception)

I will try again, and log "Exception.ToString()", hope to get more detail

[ebozduman]

@haiduong87

You wrote:

I use this GetObjectAsync public Task GetObjectAsync(string bucketName, string objectName, long offset, long length, Action cb, ServerSideEncryption sse = null, CancellationToken cancellationToken = default)

You know this method is "obsolete/deprecated". You must have seen something similar to the following warning message:

.../minio-dotnet/Minio.Functional.Tests/FunctionalTest.cs(4172,23): warning CS0618: 'MinioClient.GetObjectAsync(string, string, Action<Stream>, ServerSideEncryption, CancellationToken)' is obsolete: 'Use GetObjectAsync method with GetObjectArgs object. Refer GetObject, GetObjectVersion & GetObjectQuery example code.' [.../minio-dotnet/Minio.Functional.Tests/Minio.Functional.Tests.csproj]

Could you try replacing the old command with the recommended new method and let us know if you can still reproduce the issue?

I've run GetObjectAsync method in both recommended and the obsolete ways, but I could not reproduce the issue.

In addition, there is an open/outstanding issue in github, HttpClient The response ended prematurely #72177, and this is a possible bug in HttpClient Connection: close handling.

Please let me know if your code/script does something different than what I used to reproduce the issue; GetObject_Test1 of ../minio/minio-dotnet/Minio.Functional.Tests/FunctionalTest.cs.

[haiduong87]

@ebozduman

I will replace that Obsolete usage in our next deployment. I keep using that because I don't find the real different in source code. Technically, I will do the same:

/// <summary>
///     Get an object. The object will be streamed to the callback given by the user.
/// </summary>
/// <param name="bucketName">Bucket to retrieve object from</param>
/// <param name="objectName">Name of object to retrieve</param>
/// <param name="cb">A stream will be passed to the callback</param>
/// <param name="sse">Server-side encryption option. Defaults to null.</param>
/// <param name="cancellationToken">Optional cancellation token to cancel the operation</param>
[Obsolete(
    "Use GetObjectAsync method with GetObjectArgs object. Refer GetObject, GetObjectVersion & GetObjectQuery example code.")]
public Task GetObjectAsync(string bucketName, string objectName, Action<Stream> cb,
    ServerSideEncryption sse = null, CancellationToken cancellationToken = default)
{
    var args = new GetObjectArgs()
        .WithBucket(bucketName)
        .WithObject(objectName)
        .WithCallbackStream(cb)
        .WithServerSideEncryption(sse);
    return GetObjectAsync(args, cancellationToken);
}

Here's the piece of code of my GetObjectAsync

private static async Task GetObjectAsync(MinioClient minioClient, BucketProfile bucketProfile, string objectName, string prefix,
        Stream outputStream, RangeHeaderValue range, CancellationToken cancellationToken)
    {
        try
        {
            var formattedObjectName = Helper.FormatToCloudObjectStoragePath(prefix + objectName);
            var range1 = range?.Ranges.FirstOrDefault();
            if (range1 != null)
                await minioClient.GetObjectAsync(
                    bucketProfile.BucketName,
                    formattedObjectName, range1.From.Value,
                    range1.To.Value - range1.From.Value + 1,
                    stream => stream.CopyToAsync(outputStream, 1024 * 16, cancellationToken).Wait(cancellationToken), cancellationToken: cancellationToken
                                ).ConfigureAwait(false);

            else
                await minioClient.GetObjectAsync(
                    bucketProfile.BucketName,
                    formattedObjectName, stream => stream.CopyToAsync(outputStream, 1024 * 16, cancellationToken).Wait(cancellationToken),
                    cancellationToken: cancellationToken).ConfigureAwait(false);

        }
        catch (ObjectNotFoundException ex)
        {
            throw new FileNotFoundException(ex.Message, prefix + objectName, ex);
        }
        catch (ConnectionException ex)
        {
            if (ex.InnerException == null)
                throw;
            else
                throw ex.InnerException;
        }
        catch (MinioException ex)
        {
            if (ex.Message.Contains("The request timed-out", StringComparison.InvariantCultureIgnoreCase)) throw new TimeoutException(ex.Message, ex);

            await HandleBuckerNotFoundAsync(minioClient, bucketProfile, ex, cancellationToken).ConfigureAwait(false);
            throw;
        }
    }

I think my function is the same with the test

Thanks for the refer issue in HttpClient.

[ebozduman]

I agree @haiduong87 I also don't think replacing the deprecated method will make any difference. So, maybe not this time, but once in a while, I have seen change the behavior when the warning messages are removed, and it always feels better to do some cleanup.

[ebozduman]

@haiduong87

One more exercise/suggestion to understand if this issue is a manifestation of the original HttpClient problem addressed in 72177

72177 reports that HttpClient issue is a regression and introduced in .NET 6 and 7 preview 6 So, the first question: I assume so, but do you run against .NET 6? If so, could you repeat the reproduction steps against, let's say 5.0, and check if the issue can be reproduced. If it cannot be reproduced, we can close this issue.

[ebozduman]

Closing it per our discussion.