Provide a way to adjust the httpclient timeout for layer pushes

[baronfel]

Users aren't always in control of their network - we should provide some way for them to override the default .NET timeout for uploads (100s). This pops up in a few places, like https://github.com/Azure/azure-dev/discussions/3212.

Thoughts:

• since the user can cancel the operation at any time, should we just set an infinite timeout? If we do, we will likely need to adapt all existing callsites for the small operations to have short timeouts to prevent hangs. The Timeout in the HttpClient is used to create a linked cancellationToken that cancels after that duration, so we can orchestrate the same here - or simply use the new WaitAsync(TimeSpan) methods on Task<T> to make that super clear.
  • Note that if we use the generic WaitAsync, those throw TimeoutExceptions with little detail, so we should handle them and make them say something useful.

cc @rainersigwald @MichalPavlik

[BartNetJS]

got the same issue, see https://github.com/Azure/azure-dev/issues/3225#top

[baronfel]

This also bit @mitchdenny hard today.

[baronfel]

Customers are hitting this repeatedly on the linked issue. We may need to bump priority + backport to 8.0.200 since Aspire is using that feature branch.

cc @MichalPavlik / @rokonec / @rainersigwald

[tmds]

I wonder if the issue isn't caused by the SDK not doing chunked uploads, see https://github.com/Azure/azure-dev/discussions/3212#discussioncomment-8553439.

[baronfel]

We've got a hotfix in for 8.0.200 servicing and 8.0.300, but we should have some more usable knob. Moving this to the next milestone as a result.

[DamianEdwards]

@baronfel is a fix for this in 8.0.1xx?

[baronfel]

No, I'm not sure why I didn't backport that far originally. Folks using 1xx can use the nuget packages for a workaround now, however.

[baronfel]

I kicked a backport off that I can get servicing approval for over email.

[baronfel]

Backport is https://github.com/dotnet/sdk/pull/41294

[baronfel]

The knob seems to be usable enough for users so far. We'll close this and reopen if there is evidence of further need.