docker-elk
docker-elk copied to clipboard
deviantony
UI says: "Add a Fleet server" while the fleet extension is running
Problem description
I have installed the Fleet extension as covered by the Readme ( adding -f extensions/fleet/fleet-compose.yml to my docker compose command).
I can see the service running with docker compose ps.
Part of my confusion is that I don't grasp how the components should interact together. My assumption is that when I go to the Kiabana UI -> Management -> Fleet I should be able to enroll an agent in Fleet right away (maybe I'm wrong). But instead, the UI asks me to "Add a Fleet server" and provides the instructions to do so manually.
My understanding is that the Fleet server is already installed through the docker compose service and Kibana should not ask me to install a server.
Is that right ? And then I should start troubleshooting the issue. Or am I mistaken in my understanding and I should really install the Fleet server following the Kibana instructions ? But then what is the Fleet service in the docker compose ?
Thank you !
Extra information
Stack configuration
Docker setup
$ docker version
Client: Docker Engine - Community
Version: 28.0.1
API version: 1.48
Go version: go1.23.6
Git commit: 068a01e
Built: Wed Feb 26 10:41:12 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.0.1
API version: 1.48 (minimum version 1.24)
Go version: go1.23.6
Git commit: bbd0a17
Built: Wed Feb 26 10:41:12 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ docker compose version
Docker Compose version v2.33.1
Container logs
$ docker compose logs
(this server is under load and there are a lot of logs. Please specify which log could be useful, if any, and I will search for it)
Your assumption is correct, the server should be automatically registered and visible in Kibana like in the screenshot attached to the README. I'd suggest to start looking into the logs of the fleet-server service.
Thanks for your help.
What looks like a possible cause in the Fleet logs is this :
{"log.level":"error","@timestamp":"2025-06-11T10:06:04.284Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
I'm trying to understand why would elastic refuse the connection. So far I could not see any useful logs in elastic regarding connection issues.
I may add that I have slightly modified the compose file(s) to include Traefik in the mix in order to ease with the TLS certificates. But it should not have anything to do with internal connections between containers in my understanding. It is not fiddling with inter container connections when it comes to Kibana <-> Elastic anyway.
Now that I come to think of it, [::1], from fleet-server's perspective is itself, not elastic. And that's perfectly normal that it finds nothing on port 9200 on its own container.
So the question is, why does it try to connect to elastic through [::1] ? Where does it get that misleading information ?
From the fleet container :
root@fleet-server:/usr/share/elastic-agent# cat elastic-agent.yml
######################################
# Fleet configuration
######################################
outputs:
default:
type: elasticsearch
hosts: '${ELASTICSEARCH_HOSTS:http://elasticsearch:9200}'
$ELASTICSEARCH_HOSTS is empty so I guess it defaults to http://elasticsearch:9200.
And trying to connect to elastic from fleet does seem to work :
root@fleet-server:/usr/share/elastic-agent# curl http://elasticsearch:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","ApiKey"]}},"status":401}
You can't connect from the Fleet container without credentials, so this error is expected. By default, the Fleet extension attempts to enroll using the elastic user:
https://github.com/deviantony/docker-elk/blob/ac4a01fe60341089a474561ac587d1975ed6ba4b/extensions/fleet/fleet-compose.yml#L27-L30
Check the logs of the Fleet container, it should contain entries that can point you into the right direction. For example, it could be that the ELASTIC_PASSWORD in your .env file no longer matches the actual password for that user.
Thanks @antoineco, maybe my comments above are too verbose, I'll try to summarize them here.
The only error log in fleet is
{"log.level":"error","@timestamp":"2025-06-11T10:06:04.284Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
So basically the network connection can't be established with the elasticsearch container. Which seems normal since the fleet-server container tries to connect to [::1]:9200 which would be the fleet-server container itself, that does not host anything on port 9200. It seems to be mixing up its own container network address for the elasticsearch container address.
In order to prove that, I attach to the fleet-server container and try to curl [::1]:9200 which returns the same network error, as expected (curl: (7) Failed to connect to ::1 port 9200: Connection refused). And then I try another curl from the fleet-server container but instead of [::1]:9200 I curl http://elasticsearch:9200 which works (please ignore the credentials error, this is irrelevant for the test which is just proving that network connectivity works when using the correct address for the elasticsearch container).
So that means that somewhere in the configuration, the fleet-server container uses [::1]:9200 instead of elasticsearch:9200 to reach the elasticsearch container. And obviously that fails. I'm trying to find out where this happens and why. Any hint would be of great help. Thanks.
My bad, I missed the part about the [::1] address in your previous comments.
Maybe Elastic changed the default URL in v9, I'll need to look at the changes more closely.
Could you please try setting it explicitly, like we do on the tls branch? Simply replace https with http in the snippet below:
https://github.com/deviantony/docker-elk/blob/e4e91039e5661f74d83e86f34b11e5ae630e40fa/extensions/fleet/fleet-compose.yml#L33
Yes thanks. I need to wait until outside business hours to fiddle with the server. I will report then.
I did the upgrade to the latest commit, running 9.0.2.
I did add the ELASTICSEARCH_HOST: http://elasticsearch:9200 variable and checked within the running container that it's actually there.
The behavior is different now so I think it did something although I still see some errors trying to connect to [::1}.
Now the fleet-server container keeps restarting with exit code 1.
Here is the log. I try to understand what I could have done wrong.
fleet-server-1 | agent container initialisation - chown paths
fleet-server-1 | Requesting service_token from Kibana.
fleet-server-1 | Created service_token named: token-1749759659060
fleet-server-1 | Policy selected for enrollment: fleet-server-policy
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.049Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).startAgent","file.name":"cmd/enroll_cmd.go","file.line":677},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap p
rocess.","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.096Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":193},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":48,"agent.vers
ion":"9.0.2","agent.unprivileged":false,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.302Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source
":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.303Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":282},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"
}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.303Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":73},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.versio
n":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":79},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"
inputs":["entity-analytics","unix","logstash/metrics","mongodb/metrics","azure/metrics","rabbitmq/metrics","vsphere/metrics","cloudbeat/cis_aws","pf-elastic-symbolizer","aws-s3","filestream","iis/metrics","meraki/metrics","cloudbeat/asset_inventory_gcp","fleet-server","container","streaming","synthetics/browser",
"http/metrics","memcached/metrics","sql/metrics","zookeeper/metrics","system/metrics","pf-host-agent","linux/metrics","endpoint","audit/file_integrity","azure-blob-storage","cloudfoundry","etw","kafka","syslog","nginx/metrics","stan/metrics","aws-cloudwatch","log","elasticsearch/metrics","uwsgi/metrics","etcd/met
rics","statsd/metrics","packet","azure-eventhub","journald","unifiedlogs","synthetics/http","synthetics/icmp","cloudfoundry/metrics","mssql/metrics","haproxy/metrics","audit/auditd","cometd","http_endpoint","mqtt","oracle/metrics","jolokia/metrics","osquery","cloudbeat","docker","syncgateway/metrics","gcp/metrics
","cloudbeat/cis_azure","cloudbeat/vuln_mgmt_aws","awsfargate/metrics","cloudbeat/cis_k8s","cloudbeat/cis_gcp","cloudbeat/asset_inventory_aws","gcp-pubsub","tcp","beat/metrics","traefik/metrics","mysql/metrics","cel","gcs","lumberjack","salesforce","udp","kibana/metrics","containerd/metrics","httpjson","synthetic
s/tcp","redis/metrics","cloudbeat/asset_inventory_azure","winlog","docker/metrics","apache/metrics","nats/metrics","openai/metrics","panw/metrics","prometheus/metrics","apm","audit/system","benchmark","redis","kubernetes/metrics","postgresql/metrics","windows/metrics","activemq/metrics","kafka/metrics","netflow",
"o365audit","aws/metrics","cloudbeat/cis_eks","pf-elastic-collector"],"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilit
ies.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":85},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.ve
rsion":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.311Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":100},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agen
t.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":164},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"a
ddress":"localhost:6789","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.537Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":176},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap m
ode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.537Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":91},"message":"GRPC control socket listening at unix:///usr/share/elasti
c-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":226},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104
857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":127},"message":"Source URI changed from \"https://artifacts.elastic.co/do
wnloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.M
onitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001257f20), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"
\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.L
imit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg a
pi.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.538Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://local
host:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1450},"message":"Updating running component
model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":676},"message":"Spawned new component flee
t-server-default: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default-fleet-server: Starting: spawned pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.663Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ppid":48,"service.name":"fleet-server"
,"version":"9.0.2","commit":"5c7037cf","exe":"/usr/share/elastic-agent/data/elastic-agent-5368ce/components/fleet-server","args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"ecs.version":"1.6.0","service.ty
pe":"fleet-server","pid":68,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.663Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"
ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.666Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2
protocol has chunking enabled","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:00.666Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":696},"message":"Component state changed fl
eet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '68'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:02.051Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":843},"message":"Fleet Server - waiting for output unit","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:21:08.012Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fl
eet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.012Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-se
rver-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"Elasticsearch":{"Headers":null,"Hosts":["localhost:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"http","ProxyDisa
ble":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":null,"Timeout":90000000000},"Extra":null},"ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.012Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.012Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":536870912,"NumCounters":6400000},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-serv
er","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.
6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.013Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.na
me":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.020Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version
":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.0.2","elasticsearch_version":"9.0.2","@timestamp":"2025-06-12T20:21:08.02Z","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","ctx":"policy agent monitor","burst":1,"ecs.version":"1.6.0","service.type":"fleet-server","event_rate":4000,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.022Z","message":"Exposed over insecure HTTP; enablement of TLS is strongly recommended","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.022Z","message":"Exposed over insecure HTTP; enablement of TLS is strongly recommended","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.022Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:08.528Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.528Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-server-default (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:08.528Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->DEGRADED): Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"DEGRADED","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.035Z","message":"Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"DEGRADED","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.060Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":824},"message":"Fleet Server - Running on policy with Fleet Server integration: fleet-server-policy; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:14.060Z","log.logger":"tls","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig","file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.422Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":534},"message":"Starting enrollment to URL: http://0.0.0.0:8220/","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"ApiKey fail authentication","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"UL0abJUBZ4ny01R-6_-3","event.duration":1190712,"ecs.version":"1.6.0","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":540},"message":"1st enrollment attempt failed, retrying enrolling to URL: http://0.0.0.0:8220/ with exponential backoff (init 5s, max 10m0s)","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"HTTP request error","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"event.duration":1242339,"ecs.version":"1.6.0","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}
","error.type":"*fmt.wrapErrors","service.name":"fleet-server","service.type":"fleet-server","http.response.status_code":401,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":559},"message":"Enrollment failed: fail to execute request to fleet-server: st
atus code: 401, fleet-server returned an error: ErrUnauthorized, message: unauthorized","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"perform rollback on enrollment failure","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6
.0","service.type":"fleet-server","http.request.id":"01JXJWVJ85W7YCZ68618GDJNYB","server.address":"127.0.0.1:8220","mod":"enroll","service.name":"fleet-server","error.message":"unauthorized: apikey auth response UL0abJUBZ4ny01R-6_-3: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",
\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\", charset=\\\"UTF-8\\\"\",\"A
piKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id UL0abJUBZ4ny01R-6_-3\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\
\\"security\\\", charset=\\\"UTF-8\\\"\",\"ApiKey\"]}},\"status\":401}","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"HTTP Request","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","server.address":"127.
0.0.1:8220","http.request.method":"POST","event.duration":1271904,"service.name":"fleet-server","service.type":"fleet-server","fleet.apikey.id":"UL0abJUBZ4ny01R-6_-3","client.address":"127.0.0.1:46348","http.request.body.bytes":0,"http.version":"1.1","tls.established":false,"http.request.id":"01JXJWVJ85W7YCZ68618
GDJNYB","url.full":"/api/fleet/agents/enroll","http.response.body.bytes":69,"http.response.status_code":401,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":377},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":389},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-age
nt"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.631Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-ser
ver","service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.831Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.832Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"e
lastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:21:14.832Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: us
e of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | Error: fail to enroll: fail to execute request to fleet-server: status code: 401, fleet-server returned an error: ErrUnauthorized, message: unauthorized
fleet-server-1 | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1 | Error: enrollment failed: exit status 1
fleet-server-1 | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1 exited with code 1
I tried providing a FLEET_SERVER_SERVICE_TOKEN previously created from the kibana interface.
Before that, the fleet-server container exited with code 1 about every 15 seconds. After setting the token, it still exits with code 1 albeit a bit more gracefully, after reaching its 2 minutes timeout :
Error: fleet-server failed: timed out waiting for Fleet Server to start after 2m0s
The logs look pretty similar to me :
fleet-server-1 | agent container initialisation - chown paths
fleet-server-1 | Policy selected for enrollment: fleet-server-policy fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.206Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).startAgent","file.name":"cmd/enroll_cmd.go","file.line":677},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap p
rocess.","ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.450Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":193},"message":"Elastic Agent started","log":{"source":"elastic-agent"},"process.pid":37,"agent.vers
ion":"9.0.2","agent.unprivileged":false,"ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.662Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.InvokeWatcher","file.name":"upgrade/rollback.go","file.line":149},"message":"agent is not upgradable, not starting watcher","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.672Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":282},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"
} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.674Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":73},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.versio
n":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.715Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":79},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"
inputs":["cloudbeat/cis_eks","container","windows/metrics","aws/metrics","haproxy/metrics","kafka/metrics","sql/metrics","traefik/metrics","cel","synthetics/icmp","kibana/metrics","redis/metrics","uwsgi/metrics","cloudbeat/cis_k8s","docker","udp","unix","synthetics/http","docker/metrics","system/metrics","awsfargate/metrics","cloudbeat/cis_gcp","syslog","kubernetes/metrics","mysql/metrics","postgresql/metrics","syncgateway/metrics","nats/metrics","nginx/metrics","cloudbeat/asset_inventory_gcp","pf-elastic-collector","cloudfoundry","journald","linux/metrics","mssql/metrics","oracle/metrics","fleet-server","httpjson","lumb
erjack","etcd/metrics","openai/metrics","stan/metrics","statsd/metrics","cloudbeat/vuln_mgmt_aws","cloudbeat/asset_inventory_aws","pf-host-agent","audit/system","gcp-pubsub","gcs","http_endpoint","cloudfoundry/metrics","cloudbeat/cis_aws","redis","azure/metrics","memcached/metrics","osquery","azure-blob-storage","mqtt","elasticsearch/metrics","winlog","zookeeper/metrics","apm","pf-elastic-symbolizer","aws-s3","cometd","log","http/metrics","gcp/metrics","iis/metrics","jolokia/metrics","beat/metrics","benchmark","salesforce","tcp","filestream","synthetics/browser","activemq/metrics","prometheus/metrics","cloudbeat","cloudb
eat/cis_azure","aws-cloudwatch","azure-eventhub","o365audit","unifiedlogs","synthetics/tcp","panw/metrics","streaming","endpoint","audit/auditd","entity-analytics","mongodb/metrics","rabbitmq/metrics","vsphere/metrics","cloudbeat/asset_inventory_azure","etw","kafka","netflow","logstash/metrics","containerd/metrics","meraki/metrics","packet","audit/file_integrity","apache/metrics"],"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/capabilities.LoadFile","file.name":"capabilities/capabilities.go","file.line":48},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":85},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:41.716Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":100},"message":"Loading baseline config from /usr/share/elastic-agent/state/elastic-agent.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.045Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.NewManager","file.name":"runtime/manager.go","file.line":164},"message":"GRPC comms socket listening at localhost:6789","log":{"source":"elastic-agent"},"a
ddress":"localhost:6789","ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.056Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application.New","file.name":"application/application.go","file.line":176},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap m
ode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.060Z","log.logger":"control","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/control/v2/server.(*Server).Start","file.name":"server/server.go","file.line":91},"message":"GRPC control socket listening at unix:///usr/share/elasti
c-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","log":{"source":"elastic-agent"},"address":"unix:///usr/share/elastic-agent/state/data/Td8I7R-Zby36_zF_IOd9QVNlFblNEro3.sock","ecs.version":"1.6.0"} fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.062Z","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).Run","file.name":"runtime/manager.go","file.line":226},"message":"Starting grpc control protocol listener on port 6789 with max_message_size 104
857600","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.064Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":127},"message":"Source URI changed from \"https://artifacts.elastic.co/do
wnloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.M
onitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:true, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc001824030), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:\"\", APIKey:\"
\", SecretToken:\"\", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.L
imit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg a
pi.Config{Enabled:true, Host:\"http://localhost:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.065Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.066Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://local
host:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.067Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1450},"message":"Updating running component
model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":676},"message":"Spawned new component flee
t-server-default: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default-fleet-server: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.218Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":683},"message":"Spawned new unit fleet-ser
ver-default: Starting: spawned pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.257Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"pid":54,"service.name":"fleet-server",
"service.type":"fleet-server","version":"9.0.2","commit":"5c7037cf","ppid":37,"exe":"/usr/share/elastic-agent/data/elastic-agent-5368ce/components/fleet-server","args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-defa
ult"],"ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.258Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"
ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.264Z","log.logger":"component.runtime.fleet-server-default","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2
protocol has chunking enabled","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:42.264Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":696},"message":"Component state changed fl
eet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '54'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:43.213Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":843},"message":"Fleet Server - waiting for output unit","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:50:53.358Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"error.message":"d
ial tcp [::1]:9200: connect: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"warn","@timestamp":"2025-06-12T20:50:53.358Z","message":"Failed Elasticsearch output configuration test, using bootstrap values.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-se
rver-default"},"service.type":"fleet-server","error.message":"dial tcp [::1]:9200: connect: connection refused","output":{"Elasticsearch":{"Headers":null,"Hosts":["localhost:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"http","ProxyDisable":false,"ProxyHeaders":{},"
ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":null,"Timeout":90000000000},"Extra":null},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.359Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0
","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.359Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server
","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":536870912,"NumCounters":6400000},"ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","state":"STARTING"
,"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"recommended_mb":323
84,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":128419,"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","servi
ce.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.
6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:50:53.361Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","se
rvice.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0","ecs.version":"1.6.0"}
[...]
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.496Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.na
me":"fleet-server","service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.497Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-s
erver-default-fleet-server (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.497Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-s
erver-default (FAILED->STARTING): Starting","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING","old_state":"FAILED"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6
.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:39.498Z","message":"Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","component":{"binary"
:"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"FAILED","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","message":"Fleet Server failed","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name"
:"fleet-server","service.type":"fleet-server","error.message":"failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-
server-default-fleet-server (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"source":"elastic-agent"},"component":{"id":"fleet-serv
er-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"error","@timestamp":"2025-06-12T20:52:39.498Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":714},"message":"Unit state changed fleet-
server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: elastic fail 401: security_exception: unable to authenticate with provided credentials and anonymous access is not allowed for this request","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","
state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":377},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.
0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":389},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-age
nt"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.waitForFleetServer.func1","file.name":"cmd/enroll_cmd.go","file.line":793},"message":"Waiting for Elastic Agent to start","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","sig":"terminated
","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.212Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"flee
t-server","service.type":"fleet-server","memory_mb":128419,"recommended_mb":32384,"ecs.version":"1.6.0","ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent","file.name":"cmd/run.go","file.line":395},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"e
lastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | {"log.level":"info","@timestamp":"2025-06-12T20:52:41.413Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: us
e of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
fleet-server-1 | Error: fleet-server failed: timed out waiting for Fleet Server to start after 2m0s
fleet-server-1 | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1 | Error: enrollment failed: exit status 1
fleet-server-1 | For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html
fleet-server-1 exited with code 1
You are right, the Fleet Server is still trying to reach Elasticsearch on localhost: dial tcp [::1]:9200: connect: connection refused.
It will be difficult for me to help without a full view of the system. If it is an option for you, I'd recommend wiping Elasticsearch's data entirely and starting over, to ensure that Kibana initializes everything correctly.
Yes I have been thinking about wiping it clean. But I really can't afford to lose the data on it. Do you recommend any way to re-initialize without losing any data ? I will be looking into the snapshot / restore way. And of course I don't want to restore the settings that are causing this issue. I guess I can try to run the setup again, it should not be of any harm.
The setup won't be of any help here, but resetting the Fleet state could. I do not know how to achieve this, but maybe the Elastic forums could point you into the right direction.
As a confirmation that your issue is not caused by something wrong on the main branch, I finally got to try the setup on a fresh installation and the Fleet Server registered without any intervention from my end. I just ran the docker-elk setup, then the up command from the fleet extension's README:
On the Settings tab of Kibana's Management/Fleet view, I can see that the default output is set to http://elasticsearch:9200, as defined in Kibana's configuration:
Is your default output maybe incorrectly re-defined inside those settings?
Thanks for taking the time to test that. I'm sorry if my issue gave the impression that the main branch was at fault. Just to clarify, the first thing I did was trying a fresh install and the Fleet server worked out of the box on the main branch.
It is indeed my specific install that is at fault here. And I 'm trying to understand the underlying mechanisms in order to troubleshoot what could have gone wrong in my setup.
(I did not find the time to try to initialize the stack again as you suggested. Will try ASAP and report.)
No offense taken, the main branch could very well have been broken after the update from v8 to v9, since the tests cannot catch every issue.
Anyway, whenever you find a chance to check the Settings tab of Kibana's Management/Fleet view you should be able to find out whether you have conflicting outputs.
That was simple enough to check right away.
I see the output for fleet in the kibana.yml file is
xpack.fleet.outputs:
- id: fleet-default-output
name: default
type: elasticsearch
hosts: [ http://elasticsearch:9200 ]
is_default: true
is_default_monitoring: true
The Fleet part of the kibana config file stayed as it was by default. And that's what appears in the settings tab as well.
I'll look into re-initializing.
Trying that :
docker compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml -f extensions/filebeat/filebeat-compose.yml -f extensions/heartbeat/heartbeat-compose.yml -f extensions/fleet/fleet-compose.yml down fleet-server
[+] Running 2/2
✔ Container elk-fleet-server-1 Removed 1.0s
! Network elk_elk Resource is still in use 0.0s
Nor docker compose ps neither docker ps -a give any sign of a remaining fleet-server conatiner. I consider it gone.
Then :
docker compose -f docker-compose.yml -f extensions/metricbeat/metricbeat-compose.yml -f extensions/filebeat/filebeat-compose.yml -f extensions/heartbeat/heartbeat-compose.yml -f extensions/fleet/fleet-compose.yml up fleet-server -d
WARN[0000] mount of type `volume` should not define `bind` option
[+] Running 3/3
✔ Container elk-elasticsearch-1 Running 0.0s
✔ Container elk-kibana-1 Running 0.0s
✔ Container elk-fleet-server-1 Started 0.1s
I can see the fleet-server container up and running again through docker compose ps :
elk-fleet-server-1 elk-fleet-server "/usr/bin/tini -- /u…" fleet-server About a minute ago Up About a minute 8220/tcp
And I can see it is stuck in a restarting loop as before about every 2 minutes (corresponding to the defined timeout). And the interface in kibana is showing no sign of a fleet server.
I will take the time to dig deeper and report.
I have tried to re-install my setup on a development server, with the same settings, docker compose files ... all the same. And the fleet server starts right away and is available in the kibana UI as expected.
So I really can't tell what has gone wrong in the first place.
I will try to reset the fleet server and if nothing works then I know I can re-install the whole setup with the same settings and it should work as expected.
I tried another experiment, just for fun at this stage. I uninstalled the fleet server from the compose stack. Then I tried to install it manually following the Kibana instructions directly on the host system. It ended up giving the exact same error as the fleet server from the docker elk compose stack.
Fleet Server can't easily be installed that way unless your stack has TLS enabled and properly configured.
At least in v8, Kibana would not accept URLs with the http:// scheme, forcing you to configure the agent using a YAML manifest instead.
We were not able to identify what happened in this environment and OP confirmed that the default setup works in a fresh environment, so I'm going to close this issue for now.
Feel free to continue the discussion of course.