PerfView timeout for download symbols is too short for slow connections

[davkean]

This one bit me today, but realized that I'd run into this a bunch in the past but just worked out what caused it.

The timeout for getting a response for the server is limited to 10 seconds (or 25 seconds for \\symbols): https://github.com/microsoft/perfview/blob/d95dcb035da64f40920a41cabb850a4f5c19cb9a/src/TraceEvent/Symbols/SymbolReader.cs#L1124. This too short for my connection to the internal symbol server:

FindSymbolFilePath: Searching Symbol Server http://internalsymbolserver.
FindSymbolFilePath: In task, sending HTTP request http://internalsymbolserver/msenv.pdb/AF11E66408DD42948A0973E7ED3B68CA2/msenv.pdb
FindSymbolFilePath: Probe of http://internalsymbolserver/msenv.pdb/AF11E66408DD42948A0973E7ED3B68CA2/msenv.pdb failed: One or more errors occurred.
FindSymbolFilePath: In task, sending HTTP request http://internalsymbolserver/msenv.pdb/AF11E66408DD42948A0973E7ED3B68CA2/msenv.pd_
FindSymbolFilePath: Probe of http://internalsymbolserver/msenv.pdb/AF11E66408DD42948A0973E7ED3B68CA2/msenv.pd_ failed: One or more errors occurred.
FindSymbolFilePath: In task, sending HTTP request http://internalsymbolserver/msenv.pdb/AF11E66408DD42948A0973E7ED3B68CA2/file.ptr

FindSymbolFilePath: Waiting for initial connection to http://symweb/msenv.pdb\AF11E66408DD42948A0973E7ED3B68CA2\file.ptr.
FindSymbolFilePath: Time 10.2981382 sec.  Timeout of 10 seconds exceeded for [internalsymbolserver] .  Setting as dead server
FindSymbolFilePath: Skipping server http://internalsymbolserverbecause it was unreachable in the past, will try again in 5 min.
Trying the XX/XXYYY.PDB convention

After that marks it as a dead server and won't try again. It doesn't end up aborting the connection - so can we just let the default timeouts for WebRequest kick in?

[brianrob]

@davkean, I'm not sure why this custom logic is here, at least for symbol servers, where HttpWebRequest timeouts should kick in. There is also a path for copying via CopyStreamToFile, so I'm guessing that this code has more value there. From a quick look, it appears that the timeout should not impact you once the logic believes that the request is alive. Is this issue you're seeing that this line: https://github.com/microsoft/perfview/blob/d95dcb035da64f40920a41cabb850a4f5c19cb9a/src/TraceEvent/Symbols/SymbolReader.cs#L1046 is blocking, and so you hit the timeout before it returns?

[davkean]

Yes, I assume that its still waiting for a response. I live in Australia and my connection to the internal symbol server is pretty bad. I manually downloaded the symbols no problem.

[brianrob]

Seems reasonable to depend on HttpWebRequest, at least for the case that uses it. If you want to make a PR for this, I’d be happy to accept it. If not, no worries, and I’ll take care of it. How much is this impacting you (priority-wise)?

[davkean]

I can take a look. It looks like its from TraceEvent, can I just modify that code directly in this repostory?

[brianrob]

Yes, this repo is the authoritative source for TraceEvent as well.

[brianrob]

@davkean is this still an issue or should we close this?

[En3Tho]

Hitting this too. Can't download symbols for coreclr in 25 seconds (https://github.com/microsoft/perfview/blob/main/src/TraceEvent/Symbols/SymbolReader.cs#L1144) and get artificial error that has nothing to do with an actual web operation timeout.

I would have expected this to be configurable at least.