tidb-dashboard icon indicating copy to clipboard operation
tidb-dashboard copied to clipboard

PD profiling failed ocassionally

Open aylei opened this issue 3 years ago • 3 comments

Bug Report

Please answer these questions before submitting your issue. Thanks!

What did you do?

Profile all instances through dashboard

What did you expect to see? All profiles succeed

What did you see instead? Some profiles to PD instances failed

What version of TiDB Dashboard are you using (./tidb-dashboard --version)? v5.4.0

middle_img_v2_314c24e9-d495-478a-9607-df4524564a9g

error msg

dbaas v5.4.0 测试的时候发现 dashboard manual profiling 失败

failed to fetch and write to temp file: failed to fetch profile with *.proto format: error.pd.client_request_failed: Failed to send PD API request, cause: Get "https://db-pd-0.db-pd-peer.tidb1373933076657974810.svc:2379/pd/api/v1/debug/pprof/profile?seconds=30": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x18\x04\x00\x00\x00\x00\x00\x00\x05\x00\x10\x00\x00\x00\x03\x00\x00\x00\xfa\x00\x06\x00\x10\x01@\x00\x04\x00\x10\x00\x00"
https://staging.debug.tidbcloud.com/orgs/1369847559691367379  

aylei avatar Jan 19 '22 12:01 aylei

According to https://github.com/golang/go/issues/21336#issuecomment-325737598, I'm not sure if using the same tls.Config instance for the proxy and the dashboard client at https://github.com/tikv/pd/blob/master/pkg/dashboard/dashboard.go#L78-L88 would cause this problem.

PTAL, thanks~ @HunDunDM

shhdgit avatar Feb 23 '22 15:02 shhdgit

We should add TLS into the test cases to make sure that our features works in TLS environments. @shhdgit Could you help set them up?

breezewish avatar Feb 24 '22 00:02 breezewish

Yeah, sure. And it looks like this case needs to be tested on the pd side too.

shhdgit avatar Feb 24 '22 01:02 shhdgit