Too slow

[YorkZ]

trafficstars

It's too slow to be useful. In fact, I've never been able to see the user interface because it takes forever. I guess the reason is because it always try to compare two trees recursively? If so, maybe it should just compare one level, and compares sub directories "on demand".

[fourier]

I've been able to compare i.e. emacs source trees without significant problems. Full compare time comparable to the one in Beyond Compare. Have you tried to run it with -Q ?

[fourier]

I'll try to address it by ignoring differences in ignored files and directories, like .git, see issue #19 .

[ramestica]

very slow for me as well, even for very small directory trees. I do everything ever tramp, I wonder how bad that could be for extra slowness.

[fourier]

Have you tried to do comparison without .git directory? Just to verify if it is any better.

[YorkZ]

After almost six months, I'm coming back and give it another go. However, it's still slow. In fact, it's fast only if the directories are very small without .git directory. For my C++ source tree at work, it literally takes forever!

I've also tried emacs -Q, but no difference at all!

[fourier]

Ok, I'm trying to understand what is the difference between different configurations.

1. Which OS do you use?
2. Emacs version
3. Number of files and size of the directories
4. Remote or local
5. Is the diff tool installed ? Ztree uses 'diff' from command line.
6. What is performance comparing to say Beyond Compare on the same directories.

[YorkZ]

Sorry I should have written a bit more of the details.

1. Which OS do you use?

Linux Mint 17.1

2. Emacs version

24.5.1

3. Number of files and size of the directories

2G, 22,000 files

4. Remote or local

Local

5. Is the diff tool installed ? Ztree uses 'diff' from command line.

I don't think I would have survived without having diff in my system.

6. What is performance comparing to say Beyond Compare on the same directories

I've never used Beyond, is it a Linux application?

Hope this helps.

[fourier]

Thanks, Beyond Compare is a cross-platform commercial tool I tried to replace with this Emacs extension(this functionality - the convenient comparison of the directory trees) was what I've missed from the closed source world and decided to replicate in Emacs, to have this as a portable and window/terminal agnostic solution. Now, Ztree calls diff for every of your 22000 files since it uses it as a backend; this is why it is so slow on your machine. .git directory adds a lot to the comparison; but it still going to be slow. The branch I'm working on (with support for ignored directories like .git and other starting with dot) could help a little, but the general performance could be slow unless I could find a better solution to use as a backend.

[fourier]

@YorkZ could you please test again? Were you able to do the performance comparison to other tools which allow to merge directories?

[c02y]

Slow here, I tried twice.

Once for two different versions of tmux source code directories which contains around 500 files, 220000 lines of text it takes over 45s to load into the ztree buffer

Once for two versions of fish-shell source code directories which contains around 1400 files, 880000 lines of text it takes over 4mins to load into the ztree buffer, yes, 4 mins I wait for it for over 4mins.

I also tested them in both meld or bcompare, they are both loaded for only 1 or 2 seconds.

[YorkZ]

No, it still took forever!

Ztree calls diff for every of your 22000 files since it uses it as a backend

I'm not sure why you have to diff all the files up front? What about "lazy diffing". Also, I think because for many files, just looking at their size would tell whether they are different or not, can you only diff the files with the same size?

[fourier]

If we do not diff files upfront we cannot show the status of the directory. Idea with sizes make sense, I have it implemented in another project, and will implement in this then.

[fourier]

@c02y @YorkZ Please try with the latest version, I've added check for file sizes.

[c02y]

UPDATE:

previous version: ztree-20160925.719 current version: ztree-20161026.1249

Same tests:

Two different versions of tmux source code directories which contains around 500 files, 220000 lines of text it takes over 50s to load into the ztree buffer, much longer than the previous version of ztree.

Two versions of fish-shell source code directories which contains around 1400 files, 880000 lines of text it takes over 3m50s to load into the ztree buffer, basically the same time as the previous version of ztree

For a smaller directory, it takes more time, for the bigger directory, it takes basically the same time as the previous version of ztree.

And I notice another difference, the previous ztree will create no ztree buffer until the whole diff process completes (at least I cannot see it from the current window), the updated ztree buffer will create the ztree buffer instantly after executing the ztree-diff for two dirs, but the buffer only contains the header, the complete result of the diff will be filled into the buffer after the time as I mentioned(50s and 3m50s).

Maybe you should profiler the package and get the key point the poor performance.