high CPU consumption

[cmcm951102]

What is the current behavior?

commit "296e8162368b4bca8153e2550ec00d91982e0ae7" removed the sleep fucntion in "lib/controller/controller.py"(line 548), it cause high CPU consumption on my computer, and scan has been very slow.

What is the expected behavior?

low CPU consumption

Any additional information?

none

[shelld3v]

Are you sure that change is the root cause? How did you verify it?

[kosyan62]

I am encountering a high CPU utilization issue, as shown in the screenshot attached, with a build from the latest master branch. As suggested by @cmcm951102, I will be investigating the issue and attempting to verify their suggestion as soon as possible

[Prady18]

@cmcm951102 . It seems like the removal of the "sleep" function in the "lib/controller/controller.py" file (line 548) has caused high CPU consumption and slow scanning. The expected behavior is for the CPU consumption to be low.

[shelld3v]

@cmcm951102 . It seems like the removal of the "sleep" function in the "lib/controller/controller.py" file (line 548) has caused high CPU consumption and slow scanning. The expected behavior is for the CPU consumption to be low.

Do you have any evidence for this?

[cmcm951102]

@cmcm951102 . It seems like the removal of the "sleep" function in the "lib/controller/controller.py" file (line 548) has caused high CPU consumption and slow scanning. The expected behavior is for the CPU consumption to be low.

Do you have any evidence for this?

Wed Feb  8 11:35:13 2023    test.out

         12318916 function calls (12291255 primitive calls) in 12.588 seconds

   Ordered by: cumulative time
   List reduced from 3734 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    472/1    0.007    0.000   12.589   12.589 {built-in method builtins.exec}
        1    0.000    0.000   12.589   12.589 mytest.py:1(<module>)
        1    0.000    0.000   12.026   12.026 C:\Users\cm\Desktop\dirsearch-master\dirsearch.py:36(main)
        1    0.000    0.000   10.769   10.769 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:74(__init__)
        1    0.000    0.000   10.662   10.662 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:198(run)
        1    0.000    0.000   10.658   10.658 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:261(start)
        1    1.001    1.001   10.085   10.085 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:554(process)
        1    0.000    0.000    4.064    4.064 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:502(handle_pause)
        2    3.873    1.936    3.873    1.936 {built-in method builtins.input}
  1618990    0.973    0.000    3.866    0.000 C:\Users\cm\Desktop\dirsearch-master\lib\core\fuzzer.py:124(is_finished)
  1619015    1.388    0.000    2.894    0.000 C:\Python38-32\lib\threading.py:1071(is_alive)
    505/4    0.008    0.000    1.805    0.451 <frozen importlib._bootstrap>:986(_find_and_load)
    503/4    0.004    0.000    1.804    0.451 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
    478/5    0.004    0.000    1.802    0.360 <frozen importlib._bootstrap>:650(_load_unlocked)
    430/5    0.004    0.000    1.801    0.360 <frozen importlib._bootstrap_external>:777(exec_module)
    594/6    0.001    0.000    1.773    0.296 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
  1619014    0.694    0.000    1.260    0.000 C:\Python38-32\lib\threading.py:1017(_wait_for_tstate_lock)
        1    0.000    0.000    1.163    1.163 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:19(<module>)
  1618989    0.908    0.000    1.154    0.000 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:551(is_timed_out)
  1619350    0.761    0.000    0.761    0.000 {method 'acquire' of '_thread.lock' objects}

@shelld3v profile shows that fucntion "process" take most time. and after I add sleep fucntion back, the problem solved.

[Prady18]

@cmcm951102 . It seems like the removal of the "sleep" function in the "lib/controller/controller.py" file (line 548) has caused high CPU consumption and slow scanning. The expected behavior is for the CPU consumption to be low.

Do you have any evidence for this?

Yes

[Prady18]

@cmcm951102 . It seems like the removal of the "sleep" function in the "lib/controller/controller.py" file (line 548) has caused high CPU consumption and slow scanning. The expected behavior is for the CPU consumption to be low.

Do you have any evidence for this?

Wed Feb  8 11:35:13 2023    test.out

         12318916 function calls (12291255 primitive calls) in 12.588 seconds

   Ordered by: cumulative time
   List reduced from 3734 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    472/1    0.007    0.000   12.589   12.589 {built-in method builtins.exec}
        1    0.000    0.000   12.589   12.589 mytest.py:1(<module>)
        1    0.000    0.000   12.026   12.026 C:\Users\cm\Desktop\dirsearch-master\dirsearch.py:36(main)
        1    0.000    0.000   10.769   10.769 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:74(__init__)
        1    0.000    0.000   10.662   10.662 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:198(run)
        1    0.000    0.000   10.658   10.658 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:261(start)
        1    1.001    1.001   10.085   10.085 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:554(process)
        1    0.000    0.000    4.064    4.064 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:502(handle_pause)
        2    3.873    1.936    3.873    1.936 {built-in method builtins.input}
  1618990    0.973    0.000    3.866    0.000 C:\Users\cm\Desktop\dirsearch-master\lib\core\fuzzer.py:124(is_finished)
  1619015    1.388    0.000    2.894    0.000 C:\Python38-32\lib\threading.py:1071(is_alive)
    505/4    0.008    0.000    1.805    0.451 <frozen importlib._bootstrap>:986(_find_and_load)
    503/4    0.004    0.000    1.804    0.451 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
    478/5    0.004    0.000    1.802    0.360 <frozen importlib._bootstrap>:650(_load_unlocked)
    430/5    0.004    0.000    1.801    0.360 <frozen importlib._bootstrap_external>:777(exec_module)
    594/6    0.001    0.000    1.773    0.296 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
  1619014    0.694    0.000    1.260    0.000 C:\Python38-32\lib\threading.py:1017(_wait_for_tstate_lock)
        1    0.000    0.000    1.163    1.163 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:19(<module>)
  1618989    0.908    0.000    1.154    0.000 C:\Users\cm\Desktop\dirsearch-master\lib\controller\controller.py:551(is_timed_out)
  1619350    0.761    0.000    0.761    0.000 {method 'acquire' of '_thread.lock' objects}

@shelld3v profile shows that fucntion "process" take most time. and after I add sleep fucntion back, the problem solved.

Wasn't what I told you right

[y0sua]

Hello guys, sometimes when I leave the dirsearch run suddenly the process is ended (not completed) and I got a message "Killed", is it caused by the CPU consumption (my CPU resource is not enough)?

[Captain404]

Can confirm as soon as I run Dirsearch my CPU spikes and stays over 100% usage until I Kill dirshearch

[maurosoria]

Can you please send me your specs?

• Ram Memory and cores
• Operating system
• Python version
• SWAP file config (you can give me the output of command "free -m")

the more the better

[shelld3v]

@maurosoria I think the performance issue with dirsearch is something that exists. We should soon try to debug which part of the process that's consumes the most, and then from there try to optimize it