cloud-on-k8s TestFleetCustomLogsIntegrationRecipe is flaky

TestFleetCustomLogsIntegrationRecipe failed because the Agent isn't well configured by FleetServer because Kibana couldn't add the [Fleet Server on ECK policy] due to a 502 Bad Gateway error response from the package registry at https://epr.elastic.co/search.

https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-main/58/

=== RUN   TestFleetCustomLogsIntegrationRecipe/ES_data_should_pass_validations
Retries (30m0s timeout): ........................................
  step.go:43: 
    Error Trace:	utils.go:87
    Error:      	Received unexpected error:
     elasticsearch client failed for https://elasticsearch-42w7-es-internal-http.e2e-l3q2d-mercury.svc:9200/_data_stream/logs-elastic_agent.filebeat-default: 404 Not Found: {Status:404 Error:{CausedBy:{Reason: Type:} Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception 
     RootCause:[{Reason:no such index [logs-elastic_agent.filebeat-default] Type:index_not_found_exception}]}}
        	Test:       	TestFleetCustomLogsIntegrationRecipe/ES_data_should_pass_validations

Looking at the diagnostic, yes the .ds-logs-elastic_agent.filebeat-default-xxxx index doesn't exist.

Kibana failed to install packages and so to add the Fleet Server on ECK policy:

{"type":"log","@timestamp":"2022-02-09T19:13:03+00:00","tags":["warning","plugins","fleet"],"pid":7,"message":"Failed installing package [system] due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=system&internal=true&experimental=true&kibana.version=7.17.0]"}
{"type":"log","@timestamp":"2022-02-09T19:13:03+00:00","tags":["warning","plugins","fleet"],"pid":7,"message":"Failed installing package [elastic_agent] due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/package/elastic_agent/1.3.0]"}
{"type":"log","@timestamp":"2022-02-09T19:13:03+00:00","tags":["warning","plugins","fleet"],"pid":7,"message":"Failed installing package [fleet_server] due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=fleet_server&internal=true&experimental=true&kibana.version=7.17.0]"}
{"type":"log","@timestamp":"2022-02-09T19:13:03+00:00","tags":["warning","plugins","fleet"],"pid":7,"message":"Failed installing package [log] due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/package/log/1.0.0/]"}
{"type":"log","@timestamp":"2022-02-09T19:13:05+00:00","tags":["error","plugins","fleet"],"pid":7,"message":"Error: [Fleet Server on ECK policy] could not be added. [fleet_server] could not be installed due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=fleet_server&internal=true&experimental=true&kibana.version=7.17.0]\n    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:292:21\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:274:42)\n    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:83:7)\n    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)\n    at fleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:58:25)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"}
{"type":"log","@timestamp":"2022-02-09T19:13:05+00:00","tags":["error","plugins","fleet"],"pid":7,"message":"Error: [Fleet Server on ECK policy] could not be added. [fleet_server] could not be installed due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=fleet_server&internal=true&experimental=true&kibana.version=7.17.0]\n    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:292:21\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:274:42)\n    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:83:7)\n    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)\n    at fleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:58:25)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"}
{"type":"log","@timestamp":"2022-02-09T19:13:05+00:00","tags":["error","plugins","fleet"],"pid":7,"message":"Error: [Fleet Server on ECK policy] could not be added. [fleet_server] could not be installed due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=fleet_server&internal=true&experimental=true&kibana.version=7.17.0]\n    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:292:21\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:274:42)\n    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:83:7)\n    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)\n    at fleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:58:25)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"}
{"type":"log","@timestamp":"2022-02-09T19:13:05+00:00","tags":["error","plugins","fleet"],"pid":7,"message":"Error: [Fleet Server on ECK policy] could not be added. [fleet_server] could not be installed due to error: [RegistryResponseError: '502 Bad Gateway' error response from package registry at https://epr.elastic.co/search?package=fleet_server&internal=true&experimental=true&kibana.version=7.17.0]\n    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:292:21\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 0)\n    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:274:42)\n    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:83:7)\n    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)\n    at fleetSetupHandler (/usr/share/kibana/x-pack/plugins/fleet/server/routes/setup/handlers.js:58:25)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"}
{"type":"error","@timestamp":"2022-02-09T19:12:44+00:00","tags":[],"pid":7,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:128:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:82:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)"},"url":"https://kibana-42w7-kb-http.e2e-l3q2d-mercury.svc:5601/api/fleet/setup","message":"Internal Server Error"}

New occurence:

May 4th 2022 TestFleetCustomLogsIntegrationRecipe/ES_data_should_pass_validations https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-aks/1001/testReport

Feb 10 '22 11:02 thbkrkr

Relates to https://github.com/elastic/kibana/issues/131505.

Jul 28 '22 09:07 thbkrkr

It happened three times that night.

TestFleetCustomLogsIntegrationRecipe/ES_data_should_pass_validations x 2 https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-kind-k8s-versions/806/testReport

TestFleetMode/ES_data_should_pass_validations https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-gke-k8s-versions/786/testReport

Jul 28 '22 09:07 thbkrkr

Closing because it is a transient problem that has not reoccurred for almost 1 year.

Jun 23 '23 20:06 thbkrkr

Related to:

https://github.com/elastic/kibana/issues/131505

Jun 30 '23 07:06 thbkrkr