zaproxy icon indicating copy to clipboard operation
zaproxy copied to clipboard
verified publisher icon zaproxy

ZAP does not shutdown in command line mode if AJAX Spider fails to start the browser

Open AndreyMZ opened this issue 3 years ago • 24 comments

Describe the bug

The process hangs (does not finish) after the automation plan succeeded if the job spiderAjax is included into the plan.

Steps to reproduce the behavior

  1. Create file config.yaml with the following content: 
    --- # OWASP ZAP automation configuration file, for more details see https://www.zaproxy.org/docs/automate/automation-framework/
env:                                   # The environment, mandatory
  contexts :                           # List of 1 or more contexts, mandatory
    - name: my_context                 # Name to be used to refer to this context in other jobs, mandatory
      urls:                            # A mandatory list of top level urls, everything under each url will be included
        - "http://example.com"
jobs:
  - type: spiderAjax
  2. Run ZAP and see the output: 
    $ zap.sh -cmd -autorun "$(pwd)/config.yaml"
Found Java version 17.0.2
Available memory: 25343 MB
Using JVM args: -Xmx6335m
Job spiderAjax started
Job spiderAjax found 0 URLs
Job spiderAjax finished
Automation plan succeeded!
  3. Note that the process has not finished after the automation plan succeeded.

Expected behavior

The process has finished after the automation plan succeeded.

Software versions

OWASP ZAP
Version: 2.11.1
Installed Add-ons: [[id=alertFilters, version=13.0.0], [id=ascanrules, version=44.0.0], [id=automation, version=0.13.0], [id=bruteforce, version=11.0.0], [id=callhome, version=0.3.0], [id=commonlib, version=1.8.0], [id=diff, version=11.0.0], [id=directorylistv1, version=5.0.0], [id=domxss, version=12.0.0], [id=encoder, version=0.6.0], [id=exim, version=0.1.0], [id=formhandler, version=4.0.0], [id=fuzz, version=13.6.0], [id=gettingStarted, version=13.0.0], [id=graaljs, version=0.2.0], [id=graphql, version=0.7.0], [id=help, version=14.0.0], [id=hud, version=0.13.0], [id=importurls, version=9.0.0], [id=invoke, version=11.0.0], [id=network, version=0.0.1], [id=onlineMenu, version=9.0.0], [id=openapi, version=24.0.0], [id=pscanrules, version=39.0.0], [id=quickstart, version=33.0.0], [id=replacer, version=9.0.0], [id=reports, version=0.10.0], [id=retest, version=0.2.0], [id=retire, version=0.10.0], [id=reveal, version=4.0.0], [id=saverawmessage, version=7.0.0], [id=savexmlmessage, version=0.3.0], [id=scripts, version=30.0.0], [id=selenium, version=15.7.0], [id=soap, version=12.0.0], [id=spiderAjax, version=23.7.0], [id=tips, version=9.0.0], [id=webdriverlinux, version=33.0.0], [id=websocket, version=24.0.0], [id=zest, version=35.0.0]]
Operating System: Linux
Java Version: Private Build 17.0.2
System's Locale: en
Display Locale: en_GB
Format Locale: en_GB
ZAP Home Directory: /home/user/.ZAP/
ZAP Installation Directory: /opt/zaproxy/./
Look and Feel: Metal (javax.swing.plaf.metal.MetalLookAndFeel)

Screenshots

No response

Errors from the zap.log file

~/.ZAP/zap.log

Additional context

thread-dump.txt (received using kill -3 $(pidof java))

Would you like to help fix this issue?

  • [X] Yes

AndreyMZ avatar Mar 14 '22 07:03 AndreyMZ

Could you provide the log?

thc202 avatar Mar 14 '22 09:03 thc202

Actually, seems to be zaproxy/zap-extensions#3628. We need to release a new version.

thc202 avatar Mar 14 '22 09:03 thc202

Could you provide the log?

~/.ZAP/zap.log

AndreyMZ avatar Mar 14 '22 11:03 AndreyMZ

Thank you, there are several problems, some add-ons were not properly downloaded and the AJAX Spider is not starting (binary not found), that shouldn't cause the hang issue though.

Could you try running the plan again? The WebSockets add-on should be updated, which should fix the issue.

thc202 avatar Mar 14 '22 13:03 thc202

I have updated the addons ("WebSockets" has been updated from 24.0.0 to 25.0.0). The issue still exists.

AndreyMZ avatar Mar 14 '22 17:03 AndreyMZ

Could you provide a thread dump? e.g. kill -3 <zap_pid>

thc202 avatar Mar 14 '22 17:03 thc202

Could you provide a thread dump? e.g. kill -3 <zap_pid>

thread-dump.txt (received using kill -3 $(pidof java))

AndreyMZ avatar Mar 15 '22 08:03 AndreyMZ

Thank you, the hang is caused by Crawljax, checking if we can prevent that from the AJAX Spider add-on.

thc202 avatar Mar 15 '22 10:03 thc202

Worth noting that this only happens if there was an error when starting the browser, under normal conditions the hang should not happen.

thc202 avatar Mar 15 '22 10:03 thc202

Title updated to better reflect what the problem is.

thc202 avatar Mar 17 '22 12:03 thc202

Hi, any updates on this issue ?

adhitris avatar Oct 20 '22 02:10 adhitris

This will be fixed with #4792 (upstream fix was merged).

thc202 avatar Mar 20 '23 16:03 thc202

Can this be closed then?

kingthorin avatar Mar 20 '23 17:03 kingthorin

I was planning to close it with the same PR.

thc202 avatar Mar 20 '23 17:03 thc202

Okay, sounds good.

kingthorin avatar Mar 20 '23 17:03 kingthorin

Hey guys. As far as I understand this issue is still not fixed, is it? I had a similar issue when I finished scanning and couldn't continue to the next stage of my pipeline. This is just a test .yaml to check the work. Could you please take a look? zap.sh -daemon -autorun /zap/wrk/my.trunk_test.yaml || true

6031 [ZAP-SpiderInitThread-0] INFO  org.zaproxy.addon.spider.SpiderThread - Starting spidering scan on Context: my.trunk.context at 2023-05-16T07:48:20.579+0000
6033 [ZAP-SpiderInitThread-0] INFO  org.zaproxy.addon.spider.Spider - Spider initializing...
6040 [ZAP-SpiderInitThread-0] INFO  org.zaproxy.addon.spider.Spider - Starting spider...
6041 [ZAP-SpiderInitThread-0] INFO  org.zaproxy.addon.spider.Spider - Scan will be performed from the point of view of User: NZA
7048 [ZAP-SpiderThreadPool-0-thread-18] INFO  org.zaproxy.addon.spider.Spider - Spidering process is complete. Shutting down...
7052 [ZAP-SpiderShutdownThread-0] INFO  org.zaproxy.addon.spider.SpiderThread - Spider scanning complete: true on Context: my.trunk.context at 2023-05-16T07:48:21.598+0000
7532 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job spider found 47 URLs
7535 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job spider test of type stats failed: At least 100 URLs found [47 < 100]
7536 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job spider finished, time taken: 00:00:03
7536 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job report started
7749 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job report generated report /zap/wrk/zap_report_my_trunk.xml
7749 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Job report finished, time taken: 00:00:00
7749 [ZAP-daemon] INFO  org.parosproxy.paros.CommandLine - Automation plan succeeded!

Zavada-Nazarii avatar May 16 '23 08:05 Zavada-Nazarii

You are running ZAP in daemon mode (-daemon), which needs to be shutdown explicitly. Run ZAP in command line mode (-cmd) if you want it to exit once all work done.

thc202 avatar May 16 '23 08:05 thc202

You are running ZAP in daemon mode (-daemon), which needs to be shutdown explicitly. Run ZAP in command line mode (-cmd) if you want it to exit once all work done.

I'm trying do it also, but =>

$ zap.sh -cmd -daemon ./my.trunk_test.yaml || true
Found Java version 11.0.18
Available memory: 32013 MB
Using JVM args: -Xmx8003m
Failed due to invalid parameters: [-cmd, -daemon, ./my.trunk_test.yaml]
Command line arguments -cmd and -daemon cannot be used at the same time.
Use '-h' for more details.

Zavada-Nazarii avatar May 16 '23 08:05 Zavada-Nazarii

You removed -autorun instead of -daemon.

thc202 avatar May 16 '23 08:05 thc202

You removed -autorun instead of -daemon.

it is the same $ zap.sh -daemon -autorun -cmd /zap/wrk/my.trunk_test_ajax.yaml || true Found Java version 11.0.18 Available memory: 32012 MB Using JVM args: -Xmx8003m Failed due to invalid parameters: [-daemon, -autorun, -cmd, /zap/wrk/my.trunk_test_ajax.yaml] Command line arguments -cmd and -daemon cannot be used at the same time.

Zavada-Nazarii avatar May 16 '23 08:05 Zavada-Nazarii

You need to remove -daemon still, as the error indicates:

Command line arguments -cmd and -daemon cannot be used at the same time.

Also, note that -cmd should not be between the -autorun and the file, i.e.: zap.sh -cmd -autorun /zap/wrk/my.trunk_test_ajax.yaml || true

thc202 avatar May 16 '23 09:05 thc202

Oh yeah. It's working now, thanks, and sorry to bother you :)

Zavada-Nazarii avatar May 16 '23 09:05 Zavada-Nazarii

I'm also still seeing this issue with docker:

docker container run -v $(pwd):/zap/wrk/:rw -t softwaresecurityproject/zap-nightly zap.sh -port 50000 -dir /zap/wrk/zap-cmd -cmd -autorun /zap/wrk/.zap/zap-mc-baseline-scan.yaml

and in the thread dump:

"pool-2-thread-1" #31 prio=5 os_prio=0 cpu=1.00ms elapsed=4.77s tid=0x00007f44dc160a70 nid=0x7094 waiting on condition  [0x00007f451458a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep([email protected]/Native Method)
	at com.crawljax.core.CrawlController$1.run(CrawlController.java:101)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:539)
	at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
	at java.lang.Thread.run([email protected]/Thread.java:833)

This is preventing from integrating AF in CI/CD environment as the github action gets stuck.

Any solutions or workarounds appreciated.

megalucio avatar Nov 21 '23 09:11 megalucio

Could you share the whole zap.log file? (Obfuscated as needed.)

thc202 avatar Nov 21 '23 09:11 thc202