docker-selenium icon indicating copy to clipboard operation
docker-selenium copied to clipboard

Published 20 hours ago • verified publisher icon SeleniumHQ

[🐛 Bug]: invalid session id with selenium grid

Open amardeep2006 opened this issue 11 months ago • 4 comments

What happened?

I upgraded to selenium grid 4.18.1 this week from 4.14 and observing that some test scripts are throwing error like invalid session id: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539 Deployed on : Kubernetes Autoscaling: enabled (Deployment)

  1. The tests starts normally and goes fine for 15 to 20 minutes and then few sessions fail with this error.
  2. All the sessions do not fail at same time. In fact one of the test ran for 35 minutes successfully.
  3. I have set terminationGracePeriodSeconds: 3600 in autoscaling.
  4. It's not a session timeout case because I have captured the ingress logs and I can see https 200 status till 1 second ago then all of sudden hub deleted the session id from Map.
Sessionid-logs
  1. I also looked at the chrome-node logs and this is where chrome-node all of sudden went into draining mode ?

image

Do we know the reasons why chrome node goes into draining mode ?

Additional Info : I run around 30 tests in parallel. I run 1 browser per pod. The test duration ranges between 10 minutes to 35 minutes. Keda version previously was 2.12.0 but now it 2.13.1 with this upgrade.

Command used to start Selenium Grid with Docker (or Kubernetes)

global:
  seleniumGrid:
    # Image registry for all selenium components
    imageRegistry: internalrepo
    # Image tag for all selenium components
    imageTag: "$IMAGE_TAG_GRID"
    # Image tag for browser's nodes
    nodesImageTag: "$IMAGE_TAG_NODES"
    # Pull secret for all components, can be overridden individually
    imagePullSecret: ""
    # Log level for all components. Possible values describe here: https://www.selenium.dev/documentation/grid/configuration/cli_options/#logging
    logLevel: INFO
isolateComponents: $ISOLATE_COMPONENTS
# Basic auth settings for Selenium Grid
basicAuth:
  # Enable or disable basic auth
  enabled: $BASIC_AUTH_ENABLED
autoscaling:
  enabled: $AUTOSCALING_ENABLED
  scalingType: $SCALING_TYPE
  # Options for KEDA scaled resources (keep only common options used for both ScaledJob and ScaledObject)
  terminationGracePeriodSeconds: 3600 #default
  scaledOptions:
    minReplicaCount: 0
    maxReplicaCount: $MAX_REPLICAS_COUNT
customLabels: {"app-id": "selgrid", "app-tier": "application"}
ingress:
  # Name of ingress class to select which controller will implement ingress resource
  # Custom annotations for ingress resource
  annotations:
    cert-manager.io/cluster-issuer: "internal-acme-sectigo"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"
  # Default host for the ingress resource
  hostname: $INGRESS_DOMAIN
  tls:
    - secretName: sel-grid-tls-secret
      hosts:
        - $INGRESS_DOMAIN
router:
  imagePullPolicy: Always
distributor:
  imagePullPolicy: Always  
eventBus:
  imagePullPolicy: Always
sessionMap:
  imagePullPolicy: Always
sessionQueue:
  imagePullPolicy: Always    
hub:
  # Custom sub path for the hub deployment
  # subPath: /selenium  
  imagePullPolicy: Always
  # Resources for container
  resources:
    requests:
      memory: "2Gi"
      cpu: "3"
    limits:
      memory: "8Gi"
      cpu: "4"
  extraEnvironmentVariables:
    - name: SE_JAVA_OPTS
      value: "-Xms1024m -Xmx8192m"
chromeNode:
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for chrome-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"      
edgeNode:
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for edge-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"         
firefoxNode:
  enabled: false
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for firefox-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"

Relevant log output

Chrome Node Logs 

Date	Service	Message
2024-02-29T05:48:04.695Z	"node-chrome"	Shutdown complete
2024-02-29T05:48:04.695Z	"node-chrome"	2024-02-29 05:48:04,666 INFO stopped: xvfb (terminated by SIGTERM)
2024-02-29T05:48:03.694Z	"node-chrome"	2024-02-29 05:48:03,664 INFO stopped: vnc (terminated by SIGTERM)
2024-02-29T05:48:02.694Z	"node-chrome"	2024-02-29 05:48:02,662 INFO stopped: novnc (terminated by SIGTERM)
2024-02-29T05:48:01.694Z	"node-chrome"	2024-02-29 05:48:01,660 INFO stopped: selenium-node (terminated by SIGTERM)
2024-02-29T05:48:01.694Z	"node-chrome"	05:48:01.658 INFO [LocalNode.stopAllSessions] - Trying to stop all running sessions before shutting down...
2024-02-29T05:48:01.694Z	"node-chrome"	2024-02-29 05:48:01,656 INFO waiting for xvfb, vnc, novnc, selenium-node to die
2024-02-29T05:48:01.694Z	"node-chrome"	2024-02-29 05:48:01,655 WARN received SIGTERM indicating exit request
2024-02-29T05:48:01.694Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:48:01.694Z	"node-chrome"	Trapped SIGTERM/SIGINT/x so shutting down supervisord...
2024-02-29T05:48:00.693Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:59.692Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:58.691Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:57.691Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:56.690Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:55.689Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:54.689Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:53.688Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:51.688Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:50.687Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:49.686Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:48.686Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:47.685Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:46.684Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:45.684Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:44.683Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:43.682Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:42.682Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:41.681Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:40.680Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:38.679Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:37.678Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:36.677Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:35.677Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:34.676Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:33.675Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:32.675Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:31.674Z	"node-chrome"	Node preStop is waiting for current session to be finished if any. Node details: message: No free slots available, availability: DRAINING
2024-02-29T05:47:31.674Z	"node-chrome"	Signaling Node to drain itself
2024-02-29T05:47:31.674Z	"node-chrome"	}
2024-02-29T05:47:31.674Z	"node-chrome"	"message": "Node status was successfully set to draining."
2024-02-29T05:47:31.674Z	"node-chrome"	"value": true,
2024-02-29T05:47:31.674Z	"node-chrome"	{
2024-02-29T05:47:31.674Z	"node-chrome"	Signaling Hub to drain node
2024-02-29T05:47:31.674Z	"node-chrome"	Current Node ID is: 781bf4cf-23ae-4f5c-84b9-30a65e99eedd
2024-02-29T05:47:31.674Z	"node-chrome"	Set SE_GRID_URL internally: https://user:[email protected]
2024-02-29T05:47:31.674Z	"node-chrome"	SE_NODE_GRID_URL: https://user:[email protected]
2024-02-29T05:47:31.674Z	"node-chrome"	Detected full distributed mode: false
2024-02-29T05:29:20.910Z	"node-chrome"	05:29:20.341 INFO [LocalNode.newSession] - Session created by the Node. Id: 2e5824bb7eb7f83dc8d83774e8c7c539, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 122.0.6261.94, chrome: {chromedriverVersion: 122.0.6261.94 (880dbf29479c..., userDataDir: /tmp/.org.chromium.Chromium...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:41173}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: wss://user:password@company-se..., se:cdpVersion: 122.0.6261.94, se:vnc: wss://user:password@company-se..., se:vncEnabled: true, se:vncLocalAddress: ws://10.42.67.101:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
2024-02-29T05:29:17.909Z	"node-chrome"	Node ID: 781bf4cf-23ae-4f5c-84b9-30a65e99eedd is found in the Grid. The registration is successful.
2024-02-29T05:29:17.909Z	"node-chrome"	Grid responds a matched Node ID: 781bf4cf-23ae-4f5c-84b9-30a65e99eedd
2024-02-29T05:29:17.909Z	"node-chrome"	Node responds the ID: 781bf4cf-23ae-4f5c-84b9-30a65e99eedd with status: UP
2024-02-29T05:29:17.909Z	"node-chrome"	Set SE_GRID_URL internally: https://user:[email protected]
2024-02-29T05:29:17.909Z	"node-chrome"	SE_NODE_GRID_URL: https://user:[email protected]
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:17.387 INFO [NodeServer.lambda$createHandlers$2] - Node has been added
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:17.176 INFO [NodeServer$1.lambda$start$1] - Sending registration event...
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:17.166 INFO [NodeServer.execute] - Started Selenium node 4.18.1 (revision b1d3319b48): http://10.42.67.101:5555
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:17.164 INFO [NodeServer$1.start] - Starting registration process for Node http://10.42.67.101:5555
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:17.016 INFO [Node.<init>] - Binding additional locator mechanisms: relative
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:16.975 INFO [NodeOptions.report] - Adding chrome for {"browserName": "chrome","browserVersion": "122.0","goog:chromeOptions": {"binary": "\u002fusr\u002fbin\u002fgoogle-chrome"},"platformName": "linux","se:noVncPort": 7900,"se:vncEnabled": true} 1 times
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:16.932 INFO [NodeOptions.getSessionFactories] - Detected 2 available processors
2024-02-29T05:29:17.909Z	"node-chrome"	05:29:16.916 INFO [NodeServer.createHandlers] - Reporting self as: http://10.42.67.101:5555
2024-02-29T05:29:16.904Z	"node-chrome"	05:29:16.814 INFO [UnboundZmqEventBus.<init>] - Event bus ready
2024-02-29T05:29:16.904Z	"node-chrome"	2024-02-29 05:29:15,904 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-29T05:29:16.904Z	"node-chrome"	2024-02-29 05:29:15,904 INFO success: vnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-29T05:29:16.904Z	"node-chrome"	2024-02-29 05:29:15,904 INFO success: xvfb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-29T05:29:15.904Z	"node-chrome"	05:29:15.811 INFO [UnboundZmqEventBus.<init>] - Sockets created
2024-02-29T05:29:15.904Z	"node-chrome"	05:29:15.753 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp://mer-selgrid-selenium-hub.mer-docconv-dev-selgrid:4442 and tcp://mer-selgrid-selenium-hub.mer-docconv-dev-selgrid:4443
2024-02-29T05:29:15.904Z	"node-chrome"	05:29:15.642 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
2024-02-29T05:29:15.904Z	"node-chrome"	05:29:15.638 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
2024-02-29T05:29:15.904Z	"node-chrome"	Starting Selenium Grid Node...
2024-02-29T05:29:15.904Z	"node-chrome"	max-sessions = 1
2024-02-29T05:29:15.904Z	"node-chrome"	stereotype = '{"browserName": "chrome", "browserVersion": "122.0", "platformName": "Linux", "goog:chromeOptions": {"binary": "/usr/bin/google-chrome"}}'
2024-02-29T05:29:15.904Z	"node-chrome"	display-name = "chrome"
2024-02-29T05:29:15.904Z	"node-chrome"	[[node.driver-configuration]]
2024-02-29T05:29:15.904Z	"node-chrome"	max-sessions = 1
2024-02-29T05:29:15.904Z	"node-chrome"	drain-after-session-count = 0
2024-02-29T05:29:15.904Z	"node-chrome"	detect-drivers = false
2024-02-29T05:29:15.904Z	"node-chrome"	override-max-sessions = false
2024-02-29T05:29:15.904Z	"node-chrome"	session-timeout = "300"
2024-02-29T05:29:15.904Z	"node-chrome"	grid-url = "https://user:[email protected]"
2024-02-29T05:29:15.904Z	"node-chrome"	[node]
2024-02-29T05:29:15.904Z	"node-chrome"	port = "5555"
2024-02-29T05:29:15.904Z	"node-chrome"	[server]
2024-02-29T05:29:15.904Z	"node-chrome"	subscribe = "tcp://mer-selgrid-selenium-hub.mer-docconv-dev-selgrid:4443"
2024-02-29T05:29:15.904Z	"node-chrome"	publish = "tcp://mer-selgrid-selenium-hub.mer-docconv-dev-selgrid:4442"
2024-02-29T05:29:15.904Z	"node-chrome"	[events]
2024-02-29T05:29:15.904Z	"node-chrome"	Selenium Grid Node configuration:
2024-02-29T05:29:15.904Z	"node-chrome"	Tracing is disabled
2024-02-29T05:29:15.904Z	"node-chrome"	Setting up SE_NODE_HOST...
2024-02-29T05:29:15.904Z	"node-chrome"	Generating Selenium Config
2024-02-29T05:29:15.904Z	"node-chrome"	Appending Selenium options: --log-level INFO
2024-02-29T05:29:15.904Z	"node-chrome"	Appending Selenium options: --heartbeat-period 30
2024-02-29T05:29:15.904Z	"node-chrome"	Appending Selenium options: --session-timeout 300
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:14,833 INFO success: selenium-node entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:14,824 INFO spawned: 'selenium-node' with pid 11
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:14,822 INFO spawned: 'novnc' with pid 10
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:14,821 INFO spawned: 'vnc' with pid 9
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:14,819 INFO spawned: 'xvfb' with pid 8
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:13,816 INFO supervisord started with pid 7
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:13,816 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:13,815 INFO RPC interface 'supervisor' initialized
2024-02-29T05:29:15.904Z	"node-chrome"	2024-02-29 05:29:13,812 INFO Included extra file "/etc/supervisor/conf.d/selenium.conf" during parsing


=========================================================================
Hub Logs 

Date	Service	Message
2024-02-29T05:48:09.502Z	"hub"	05:48:08.603 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "9a964da180f75a84a82f6cef79186cd7","eventTime": 1709185688603186322,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "DELETE","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:09.502Z	"hub"	05:48:08.603 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.502Z	"hub"	05:48:08.603 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.502Z	"hub"	05:48:08.585 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "eb7e4b1210e589c8cd9328dac07397de","eventTime": 1709185688584176611,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "DELETE","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:09.502Z	"hub"	05:48:08.584 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.502Z	"hub"	05:48:08.584 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.502Z	"hub"	05:48:08.562 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "94e3ae6622f508ea65432ab4703f38d8","eventTime": 1709185688561140756,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "GET","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539\u002fscreenshot","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:09.502Z	"hub"	05:48:08.562 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.502Z	"hub"	05:48:08.562 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.501Z	"hub"	05:48:08.543 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "23048d081e3dbd65821bb8ce1927ba36","eventTime": 1709185688542166654,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "GET","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539\u002fscreenshot","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:09.501Z	"hub"	05:48:08.543 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.501Z	"hub"	05:48:08.543 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.501Z	"hub"	05:48:08.516 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "1fa9ab2c5a4ae24404e8eed2f3db7b53","eventTime": 1709185688516251615,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "POST","http.request_content_length": "91","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539\u002felement","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:09.501Z	"hub"	05:48:08.516 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:09.501Z	"hub"	05:48:08.516 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:08.500Z	"hub"	05:48:08.498 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "45422d9e436bb06548884e547c4c36c0","eventTime": 1709185688497267736,"eventName": "exception","attributes": {"exception.message": "Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539Build info: version: '4.18.1', revision: 'b1d3319b48'System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.4.0-170-generic', java.version: '11.0.22'Driver info: driver.version: unknown\tat org.openqa.selenium.grid.sessionmap.local.LocalSessionMap.get(LocalSessionMap.java:132)\tat org.openqa.selenium.grid.sessionmap.SessionMap.getUri(SessionMap.java:84)\tat org.openqa.selenium.grid.router.HandleSession.lambda$loadSessionId$4(HandleSession.java:223)\tat io.opentelemetry.context.Context.lambda$wrap$2(Context.java:224)\tat org.openqa.selenium.grid.router.HandleSession.execute(HandleSession.java:180)\tat org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:383)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.router.Router.execute(Router.java:87)\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.grid.security.BasicAuthenticationFilter.lambda$apply$0(BasicAuthenticationFilter.java:54)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:346)\tat org.openqa.selenium.remote.http.Route.execute(Route.java:69)\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:63)\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)","exception.type": "org.openqa.selenium.NoSuchSessionException","http.client_ip": "10.181.103.177","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.router.HandleSession","http.host": "company-selgrid-np-exp.int.np.bed.k8s.com","http.method": "POST","http.request_content_length": "91","http.scheme": "HTTP","http.target": "\u002fsession\u002f2e5824bb7eb7f83dc8d83774e8c7c539\u002felement","http.user_agent": "webdriver\u002f7.33.0","session.id": "2e5824bb7eb7f83dc8d83774e8c7c539"}}
2024-02-29T05:48:08.500Z	"hub"	05:48:08.498 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:08.500Z	"hub"	05:48:08.498 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:02.492Z	"hub"	05:48:01.671 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:48:02.492Z	"hub"	05:48:01.671 INFO [GridModel.release] - Releasing slot for session id 2e5824bb7eb7f83dc8d83774e8c7c539
2024-02-29T05:29:20.786Z	"hub"	05:29:20.348 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 2e5824bb7eb7f83dc8d83774e8c7c539

Operating System

kubernetes 1.23.14

Docker Selenium version (image tag)

4.18.1

Selenium Grid chart version (chart version)

0.28.1

amardeep2006 avatar Feb 29 '24 07:02 amardeep2006

@amardeep2006, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

github-actions[bot] avatar Feb 29 '24 07:02 github-actions[bot]

Can you also check the logs of keda-operator at that time, to see if any deployment was scaled down by the Scaler. When looking at your logs, it's similar to an issue still open for discussion around autoscaling here https://github.com/SeleniumHQ/docker-selenium/issues/2129

VietND96 avatar Feb 29 '24 10:02 VietND96

Thanks @VietND96 for response. I looked at the keda operator logs and it's full of unrelated errors. It's failing to get firefox scaled object because I have set firefox:enabled as false. May be this is another area of improvement in helm chart(unrelated to this issue). I am using chrome nodes.

image

image

I looked at the issue you mentioned and it seems like almost my issue. I have tried disabling autoscaling and all tests worked fine so the issue is related to autoscaling that's for sure. What to you suggest as further troubleshooting ?

  1. Shall I downgrade keda to 2.12.0 ?
  2. Shall I consider jobs strategy instead of HPA ?

I have another questing : Looking at the chrome-node logs , what do you think ? Let's assume KEDA operator instructed for downscaling, Why did not the preStop script bale to hold the DRAINING node till tests was complete ? Can there be some bug in prestop script logic ?

edit: I just saw one more issue that looks like similar in nature. https://github.com/SeleniumHQ/docker-selenium/issues/2155

amardeep2006 avatar Feb 29 '24 11:02 amardeep2006

I will share further details tomorrow as I feel it could be kubernetes killing pods because of many reasons

  1. disk pressure
  2. cpu/ram crunch on underlying k8s node
  3. Health check failure , is livenessProbe enabled by default in HPA ? The most probable reason looks to me as follows:
  4. I also see a default config chromeNode.terminationGracePeriodSeconds=30 . Can it be playing some role here ?? exactly after 30 seconds I see SIGTERM in logs. Should not it be inherited from my value file as 3600 ?

image

amardeep2006 avatar Feb 29 '24 18:02 amardeep2006

@VietND96 I applied the chromeNode.terminationGracePeriodSeconds=3600 setting and the issue is disappeared. I see following issues that may need fix in helm chart:

  1. Global terminationGracePeriodSeconds is not applied to individual pods.
  2. Only Global Autoscaling setting are applied to each browser pods. I tried to set below settings but HPA still applies global configs and sets max replica 40 and not 3.
firefoxNode:
  enabled: true
  imagePullPolicy: Always
  # /dev/shm volume
  dshmVolumeSizeLimit: "2Gi"
  # Resources for firefox-node container
  resources:
    requests:
      memory: "1Gi"
      cpu: "1"
    limits:
      memory: "2Gi"
      cpu: "2"
  extraEnvironmentVariables: 
    # - name: "SE_VNC_NO_PASSWORD"
    #   value: "1"
    - name: "SE_VNC_VIEW_ONLY"
      value: "1"         
  autoscaling:
    scaledOptions:
      minReplicaCount: 0
      maxReplicaCount: 3
  terminationGracePeriodSeconds: 3600

image

edit : Downside is few pods may live for 3600 seconds in terminating state and still do processing. I can live with that for now.

amardeep2006 avatar Mar 01 '24 09:03 amardeep2006

Ok, let me check any regression broken, since in README I updated that default 3600 will be applied for all nodes if there is no individual config override

VietND96 avatar Mar 01 '24 10:03 VietND96

It was a defect actually, the logic is handled but the template name is not called in Node spec YAML, so value 30 in each node is picked up directly. I just fixed it and added a template test to verify it. I am considering other fixes before bumping a new chart version soon.

VietND96 avatar Mar 02 '24 05:03 VietND96

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

github-actions[bot] avatar Apr 02 '24 00:04 github-actions[bot]