jbruchon
How to build on Windows? Why are dedupe and symlink turned off by default?
I haven't seen any instructions about compiling jdupes on Windows. I have tried mingw64's make with the following command:
mingw32-make.exe CFLAGS_EXTRA=-DON_WINDOWS
but it ends with
expr: syntax error: unexpected argument '5'
process_begin: CreateProcess(NULL, uname -s, ...) failed.
mingw32-make: Makefile:103: pipe: Bad file descriptor
process_begin: CreateProcess(NULL, uname -s, ...) failed.
mingw32-make: Makefile:112: pipe: Bad file descriptor
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -DON_WINDOWS -c -o win_stat.o win_stat.c
process_begin: CreateProcess(NULL, cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -DON_WINDOWS -c -o win_stat.o win_stat.c, ...) failed.
make (e=2): Nie mo┐na odnalečŠ okreťlonego pliku.
mingw32-make: *** [<builtin>: win_stat.o] Error 2
I assume it errors this way because the Makefile uses the Unix-only commands, like
GCCVERSION = $(shell expr gcc -v 2>&1 | grep 'gcc version ' | cut -d\ -f3 | cut -d. -f1 \>= 5)
Is that a bug or am I trying to build it the wrong way?
By the way, why have you disabled dedupe and symlink features in the default builds from the releases page? If they were enabled by default, I wouldn't need to compile it myself :)
You need to use MSYS2 with MinGW-w64 installed. You do not need to pass anything special to make to build on MSYS2 + MinGW-w64. The Makefile will detect a Windows NT environment and automatically make any necessary changes. Remember that "disabling" certain features in cross-platform software development can be for the purpose of not building code or performing tasks that are not supported or implemented.
Dedupe support isn't disabled; it's not supported on Windows and never will be. ReFS supports the required cloning functionality but Microsoft removed ReFS from all non-enterprise versions of Windows 10. ReFS also just plain sucked for a variety of reasons detailed in that same article; I only mention it for the sake of completeness. I will not add ReFS cloning support to jdupes.
Symlink support isn't disabled; it's not implemented on Windows, but it could be with some work (that I don't currently want to do). Symlinks on Windows can be dangerous (most Windows software is not symlink-aware, etc.) and before a certain revision of Windows 10 (including Windows 8.x, 7, and Vista) they required administrative elevation to create. I don't feel like rewriting the symlink functionality to use backslashes, but that's more me being lazy than any practical concern. There is also the issue of the missing lstat() call in the MinGW runtime, so I'd have to write some workaround code to implement lstat() as well.
Here's a successful build on my machine with MSYS2's MinGW-w64 (-j12 runs 12 operations in parallel since I have a Ryzen 7 3700X 16-core processor):
Owner@computer MINGW64 /c/github/jdupes
$ make -j12
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o win_stat.o win_stat.c
./tune_winres.sh
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jdupes.o jdupes.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jody_paths.o jody_paths.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jody_sort.o jody_sort.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jody_win_unicode.o jody_win_unicode.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jody_strtoepoch.o jody_strtoepoch.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o string_malloc.o string_malloc.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o oom.o oom.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o jody_cacheinfo.o jody_cacheinfo.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o act_deletefiles.o act_deletefiles.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o act_linkfiles.o act_linkfiles.c
1.20.2 = 1,20,2,0 (1.20.2.0)
windres winres.rc winres.o
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o act_printmatches.o act_printmatches.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o act_summarize.o act_summarize.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o act_printjson.o act_printjson.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -c -o xxhash.o xxhash.c
cc -Wall -Wwrite-strings -Wcast-align -Wstrict-aliasing -Wstrict-prototypes -Wpointer-arith -Wundef -Wshadow -Wfloat-equal -Waggregate-return -Wcast-qual -Wswitch-default -Wswitch-enum -Wconversion -Wunreachable-code -Wformat=2 -std=gnu99 -O2 -g -D_FILE_OFFSET_BITS=64 -fstrict-aliasing -pipe -DSMA_MAX_FREE=11 -DNO_ATIME -Wextra -Wstrict-overflow=5 -Winit-self -DNDEBUG -municode -D__USE_MINGW_ANSI_STDIO=1 -DON_WINDOWS=1 -o jdupes win_stat.o winres.o jdupes.o jody_paths.o jody_sort.o jody_win_unicode.o jody_strtoepoch.o string_malloc.o oom.o jody_cacheinfo.o act_deletefiles.o act_linkfiles.o act_printmatches.o act_summarize.o act_printjson.o xxhash.o
Thanks to your post, I've updated the INSTALL document to cover Windows build requirements and to explain why dedupe and symlinks aren't supported. Thanks for your contribution!
Thank you very much for your answer. I use symlinks very often, so it's unfortunate that it is not implemented in the Windows build. It would be great to be able to handle duplicates across multiple drives. For instance, use hardlinks when it is the same filesystem and use symlinks when it is a different filesystem.
As far as ReFS is concerned - I get your point but things have improved a lot. ReFS since version 3.5 supports hardlinks and, as a result, can be used as a system partition. I have seen screenshots from people who have managed to run Windows from ReFS volumes. However, Windows 11 is the only consumer OS that offers support for ReFS 3.5 or higher (it comes with 3.7 support).
You can also bypass the Enterprice/Server restriction and use ReFS on any version on Windows by booting into the Windows Recover Environment and format the partition as ReFS from there. It will succeed. Then you can normally boot to Windows and use the ReFS volume normally as only ReFS formatting has been disabled in non-server editions. Writing to already formatted volumes works fine :)
The only thing that I miss from NTFS is compression, but hopefully that will be added in the future. Sorry for that little offtopic :)
I found out that you can format as ReFS with these directions without having to boot into WinPE. I'm looking into adding symlink support now.
Thank you for the link and for changing your mind about making symlinks available in Windows build. I have read various pros and cons of symlinks handling in Windows and I get your point. However, it exists and it is commonly used (especially by developers and power users) and using them as a cross-filesystem dedupe mechanism seems like an obvious choice.
Of course, jdupes would need to be run elevated in order to use the symlinks feature or the Windows policy that enables regular users to create symlinks would need to be turned on. Thank you again for considering adding this feature - it would be a huge timesaver.
One more thing I would ask you to consider is some kind of hybrid mode where jdupes uses hardlines if possible and symlinks otherwise. Maybe it is already working when -L and -l switches are combined, I wasn't able to test since, as you said, symlink support is not present in the Windows build of jdupes
I've implemented most of what's needed to do this, but I am having to fight with my relative path generation code, so it may be next year before I can get it working. I would point out that another danger of symlink-based deduplication is that if you delete the link target, the symlink remains but the data is lost. You may also be interested in knowing that symlinks are how Windows Home Server would pool disks: files would be kicked to other disks and linked, presenting one big volume composed of a bunch of ad-hoc added disks. The problem is that if something happened to a disk, you'd lose whatever happened to end up there, but the symlinks would remain.
I have actually considered adding the ability to fall back on a different action when the primary action fails. A lot of code in the action C files is redundant and behaviorally inconsistent. A major rewrite of them is needed, but it's not a priority.
I look forward to seeing symlink support on Windows as well, even if that support comes with some warnings or special considerations. There really is no other cross-platform tool like jdupes that offers all of its functionality and speed while still being trustworthy with handling ones files. Any other tools which one may attempt to use, to make up for jdupes lack of symlink support on Windows, are severely lacking in one way or another. It would be great to have symlink functionality on Windows as it would finally make jdupes the only dupe-finder/dedupe tool one needs, regardless of platform.
Obviously, this is easy for someone like me to say.. while implementing it is a much larger task. So ya, definitely no pressure but hopefully @jbruchon finds the time/motivation to implement this feature at some point. However, given what an excellent utility he has already provided us, this would obviously just be the cherry on top.
Edit: Just went to send you a small donation on ko-fi.com and came across a lot of your other content (including your article on forking/optimizing fdupes and your YouTube content). Must say, I'm very impressed! Just subscribed to your YouTube channel.
