`azure-storage-blob` recorded tests hanging on `pypy3.9`

[scbedd]

azure-storage-blob is randomly hanging on ubuntu 2004 pypy3.0. That I have seen in investigations so far, there were no new package releases that caused this behavior.

Need to check (or recheck):

• package dependencies

• agent version

• pypy version?

And add upload of the tox logs even in the case of a build timeout. This is a big one to actually resolve this going forward.

[scbedd]

~~Narrowed to specific test failure~~

~~It looks like we're timing out because we're getting a TON of AddSanitizer failures on each run. The failure to add sanitizer results in slow startup of the tests, then weird 404 errors in the tests as sanitizers aren't active where we expect them to be. This means that we start getting mismatch errors because headers like Content-Length aren't being cleared out.~~

Looks like a real mismatch error

At least at the outset it does. It feels like it's matching to a different request almost.

[scbedd]

I've done a bunch more digging. After isolating to a single test + working and non-working version, this starts to become a bit clearer.

On this raw log search the content Request starting HTTP/1.1 PUT http://localhost:5000/utcontainer28633a27?restype=container - 0. It matches twice. The second request fails because we've already matched it. If you look at the raw log for a working version of python this request only shows up once.

So now we understand the mechanics of the failure, but we don't understand yet why the request is firing twice.

[scbedd]

Need to write up exact repro details for @jalauzon-msft , he created an env using anaconda and saw no issues. Maybe this has to do with the the raw python versions we're using?

EDIT: I shared via gist here.