Stack overflow on Windows at the end of scanning

[e-gaulue]

Hi all,

This is a separate issue of #130.

This happen on a 60 Go drive with around 320 000 items. Max directory depth is 3. jdupes-loud.exe don't give so much idea regarding the problem. It just stops at the end of the scan. Last lines give:

grokdir: scanning '.\ZHulk\Comm203901' (order 1, recurse 1)
getdirstats('.\ZHulk\Comm203901', 00000000005ff498, 00000000005ff48c)
traverse item right: 952593295:465376
travdone_alloc(281474977176032, 952593295)
travdone_alloc returned 000000001a12fda0
FindFirstFile: .\ZHulk\Comm203901
Loop start
grokdir: readdir: '.'
grokdir: readdir: '..'
grokdir: readdir: 'Documents.pdf'
init_newfile(len 45, filelistp 000000014002e0f8)
check_singlefile: checking '.\ZHulk\Comm203901\Documents.pdf'
getfilestats('.\ZHulk\Comm203901\Documents.pdf')
check_singlefile: extfilter check: 00000004 32188 3000 .\ZHulk\Comm203901\Documents.pdf
check_singlefile: all checks passed

This is the last file of the last dir of this disk. And the number of scanned items is the same than the one reported by Windows.

Other information:

• We got filename with accent (don't think it's coming from this)
• We certainly have fully filled 1023 hard links for some files
• Last version known to work is 1.13 and stack overflow issue begins with 1.13.1.

Example using 1.19 version:

cd i:
C:\Users\myaccount\Desktop\jdupes-1.19.0-win64\jdupes.exe -L -r -X size+:3kb .

gives and then

I'm ready to help doing any test anyone can think of.

Best regards,

[jbruchon]

The only actual code change in 1.13.1 was fixing an incorrect null pointer check that resulted in a tree not being freed properly. See if the problem goes away with -U and let me know. (Don't use -L when you test it, just to be safe.)

[e-gaulue]

With -U, scanning is super-fast and when finished then dupes analyse is working:

Can I use -U with -L with no risk?

[jbruchon]

As long as you don't have any symbolic links or junctions that point to a directory AND you don't specify overlapping directory trees on the command line, you can do that without risk. The protection that -U disables is the double-traversal prevention tree. It records every directory seen so far and prevents recursion into ones that are already loaded. It solves a problem that actually still exists in fdupes where you could go fdupes -r same_dir same_dir and it would cause every file inside to match itself.

[e-gaulue]

I confirm:

cd i:
C:\Users\myaccount\Desktop\jdupes-1.19.0-win64\jdupes.exe -U -L -r -X size+:3kb .

works well.

I don't know what is the problem with the null pointer check. I: is a typical NTFS local volume.

Regards,

[jbruchon]

I'll run some tests with valgrind on Linux and see what's going on. There may be some corruption happening that is being covered up by string_malloc

[jbruchon]

FYI I did find out what the problem is. Same as when you reported the other overflow. There's recursion in the code that frees the travdone data. I'm working on changing it over to a hash table now, which should eliminate recursion and make it faster as well.

[jbruchon]

@e-gaulue Please test with current Git master and let me know if you still have issues.

[e-gaulue]

I have time to test it theses days. May I have a windows binary? Master is not released.

[jbruchon]

I'm going to assume that this is no longer an issue after the stack size increases.