aws-xray-sdk-go icon indicating copy to clipboard operation
aws-xray-sdk-go copied to clipboard

Does the SDK respect DNS TTL?

Open rclark opened this issue 4 years ago • 4 comments

2020-11-10T16:14:13Z [DEBUG] Error occurred while refreshing sampling rules. RequestError: send request failed
caused by: Post "http://172.31.133.144:443/GetSamplingRules": dial tcp 172.31.133.144:443: i/o timeout

I have a container that's been logging this error for the last hour or so. 172.31.133.144 is a stale IP address for the DNS name of my daemon service, which was used in the xray.Configure() step when the container launched. That DNS record has a low TTL (60s). Does the SDK respect and re-resolve DNS when TTL periods expire?

rclark avatar Nov 10 '20 17:11 rclark

Hi @rclark ,

Do you think SDK is using 172.31.133.144 to send data to X-Ray Daemon? I don't think SDK respects DNS TTL in the scenario you mentioned. Why this ip 172.31.133.144 becomes stale? Would you be able to explain how you're using X-Ray SDK with X-Ray daemon setup?

bhautikpip avatar Nov 23 '20 23:11 bhautikpip

I'm running the xray daemon on a set of ECS tasks, using AWS CloudMap as the discovery layer. This means I have a VPC-internal multivalue DNS record that maps a domain name to a set of live containers. Those ECS containers can stop for any number of reasons. New ones come online and the CloudMap-managed DNS record updates to reflect the current state.

rclark avatar Nov 24 '20 17:11 rclark

Yes, @rclark I think the part of the reason you're seeing such logs could be because when new ECS containers spins up SDK still used the stale IP address that you had. As far as I know SDK doesn't support anything like this. This can be added to SDK as an enhancement. I will add this task in our backlog but meanwhile we do appreciate PRs :)

bhautikpip avatar Dec 14 '20 22:12 bhautikpip

Thanks for the info.

If my application runs xray.Configure() more than once, would that re-resolve the DNS name to a new IP address each time I run it?

rclark avatar Dec 15 '20 00:12 rclark