hashdeep
hashdeep copied to clipboard
jessek
poor windows performance
with the latest release (4.3) on a windows 7 64bit system, I get terrible performance with this command
md5deep64 -l -r .
resource manager shows md5deep is reading at about 21 MB/sec (when the disk should be able to read at 100MB) and the system seems sluggish and unresponsive (even though it only uses ~ 5% of cpu)
but if I use old version 3.9.2, using the 32bit version of the .exe, I get expected performance (~80MB/sec), everything is normal.
also, I noticed the .exe file for 4.3 is 965 KB yet version 3.9 was only 59 KB, that seems like a very large difference, I was worried perhaps it had a virus or Trojan at first due to the huge size difference.
Based on your email, my guess is that you are running on a multi-core system and that the multiple threads are competing for the disk. Can you try re-running and add a "-j1" or "-j0" argument and see what that does? I thought that we had disabled multi-threaded by default on md5deep on Windows, as the OS seems to be unable to properly prioritize concurrent I/O requests, but it doesn't seem that patch made it in to the release. If so, then we will commit a 1-line change to select a single thread when md5deep (in particular) is running on Windows. Since sha1 runs 3x slower than md5, it is likely that you would still want multi-threading with sha1deep.
Looking forward to hearing from you.
I am having this exact same issue. Using the following helps:
"c:\Users\rbeede\Software\md5deep-4.3\md5deep64.exe" -r -j 0 . (I also tried -j0 for good measure with the same result)
I am down to it only accessing 3 files at the same time, but disk performance is still getting killed.
I would have expected -j 0 to force it to only process 1 file at a time.
Is there a way to force it to only process one file at a time and not open any other files at the same time? Maybe a -j -1 option value for this ?
Did you try -j1?
On Jul 31, 2013, at 9:12 PM, Rodney Beede [email protected] wrote:
I am having this exact same issue. Using the following helps:
"c:\Users\rbeede\Software\md5deep-4.3\md5deep64.exe" -r -j 0 . (I also tried -j0 for good measure with the same result)
I am down to it only accessing 3 files at the same time, but disk performance is still getting killed.
I would have expected -j 0 to force it to only process 1 file at a time.
Is there a way to force it to only process one file at a time and not open any other files at the same time? Maybe a -j -1 option value for this ?
— Reply to this email directly or view it on GitHub.
Also, what is your hardware? Are you running on a VM or on native hardware? How many cores? What's the CPU?
On Jul 31, 2013, at 9:12 PM, Rodney Beede [email protected] wrote:
I am having this exact same issue. Using the following helps:
"c:\Users\rbeede\Software\md5deep-4.3\md5deep64.exe" -r -j 0 . (I also tried -j0 for good measure with the same result)
I am down to it only accessing 3 files at the same time, but disk performance is still getting killed.
I would have expected -j 0 to force it to only process 1 file at a time.
Is there a way to force it to only process one file at a time and not open any other files at the same time? Maybe a -j -1 option value for this ?
— Reply to this email directly or view it on GitHub.
I tried -j1 with the same effect as -j 0. It helps versus -j 4, but it still opens multiple files at the same time.
My specs: Windows 7 64-bit Professional (all sp and patches) No VM. Physical hardware Gigabyte motherboard AMD Phenom (4 cores) 3.5GHz 8GB Ram 2TB HDD
Thanks for the follow-up. How do you know that it's opening multiple files at the same time. Is it reading multiple files at the same time as well? This is most odd.
Would you be willing to try out a beta version of 4.4? I've posted it to http://jessekornblum.com/tmp/
thanks!
Jesse Kornblum [email protected]
I open Resource Monitor Disk tab Disk Activity pane
The md5deep process shows multiple files open with read activity.
It is easiest to reproduce when doing a directory with several files of 1GB+ size.
I will try the beta version.
The beta version run with "md5deep64.exe -r -z -c -j 1 ." works as expected and only opens 1 file at a time.
Sure. What was changed in 4.4? On Aug 1, 2013, at 12:31 PM, Jesse Kornblum [email protected] wrote:
Would you be willing to try out a beta version of 4.4? I've posted it to http://jessekornblum.com/tmp/
thanks!
Jesse Kornblum [email protected] — Reply to this email directly or view it on GitHub.
A lot has changed in 4.4, but I'm not sure exactly what. The last release was in October 2012. Everything we've done to the codebase since then, including the changes to how threading is done, has yet to be released. My plan is to clean things up this week and publish/announce on Monday 12 Aug.