Elastic-Agents dropped and now can't enroll any to Fleet

[pc-mrousseau]

Hello Team - We've observing an issue similar to https://github.com/elastic/beats/issues/25430 using Elastic Cloud v7.13.0.

# elastic-agent enroll -d --url=https://(foo).fleet.us-west-2.aws.found.io:9243 --enrollment-token=(bar)==
The Elastic Agent is currently in BETA and should not be used in production

2021-07-06T12:23:30.225-0700	WARN	cmd/enroll_cmd.go:370	Remote server is not ready to accept connections, will retry in a moment.
2021-07-06T12:24:30.225-0700	INFO	cmd/enroll_cmd.go:377	Retrying to enroll...
2021-07-06T12:24:30.328-0700	WARN	cmd/enroll_cmd.go:370	Remote server is not ready to accept connections, will retry in a moment.
2021-07-06T12:26:30.328-0700	INFO	cmd/enroll_cmd.go:377	Retrying to enroll...

Based on the enroll code it should be ErrConnRefused however we can totally connect to it, as tested by curl:

# curl https://(foo).fleet.us-west-2.aws.found.io:443
{"ok":false,"message":"Unknown resource."}

Tried the same with port 9243 and same results. We also tried terminating the Fleet instances and restarting new ones.

Tried this with both 7.13.0 and 7.13.2 agent, using the RPM install + enroll path.

# elastic-agent version
Binary: 7.13.0 (build: 054e224d226b42a1dd7c72dcf48c3f18de452e22 at 2021-05-20 00:57:04 +0000 UTC)
Daemon: 7.13.0 (build: 054e224d226b42a1dd7c72dcf48c3f18de452e22 at 2021-05-20 00:57:04 +0000 UTC)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[pc-mrousseau]

As an update to this issue - we worked with Elastic to confirm that this is related to a known bug at the Cloud Elastic Cluster level where the Fleet servers do not play well with IP filtering rules applied. If you remove all of the Cloud IP filtering it works again. This was identified as potentially fixed in 7.14 (which is now out). I was not able to confirm this yet.

[ruflin]

It was indeed the plan to get this fixed in 7.14. @michalpristas can you confirm?

[zez3]

I'm on 7.15 and still see this

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!