duc index may hang?

[skruseman]

Running duc 1.3.3 from a Docker container running Ubuntu 16.04 at some point seemed to hang: the job never ended but accoring to top the process still used cpu cycles. Also sudo lsof | grep duc | grep <pattern in the dir tree being indexed> did not change any longer. Nothing seemed wrong with dirs and files (thru NFS) in question. I'm interested in what conditions could make the indexing hang: e.g. files being altered/appended to/added/deleted?

[zevv]

Quoting skruseman (2020-03-09 13:54:49)

Running duc 1.3.3 from a Docker container running Ubuntu 16.04 at some point seemed to hang: the job never ended but accoring to top the process still used cpu cycles. Also sudo lsof | grep duc | grep <pattern in the dir tree being indexed> did not change any longer. Nothing seemed wrong with dirs and files (thru NFS) in question. I'm interested in what conditions could make the indexing hang: e.g. files being altered/appended to/added/deleted?

No idea, I have not seen this happen. can you attach strace to the process and see what is happening?

$ strace -p

-- :wq ^X^Cy^K^X^C^C^C^C

[skruseman]

I already killed that run. Will do on any next run that behaves in same way. Also, I've created a second container running Ubuntu 18.04 and duc 1.4.3 and prefer to run that more recent one.

[l8gravely]

skruseman> I already killed that run. Will do on any next run that skruseman> behaves in same way. Also, I've created a second container skruseman> running Ubuntu 18.04 and duc 1.4.3 and prefer to run that skruseman> more recent one.

Why are you bothering to run in a container? Just curious. Since duc needs privs to look at all the files in volume, assuming they are owned by various different users, it really wants to be run as root.

And as Ico says, get a trace of the hung process so we can look it over.

John

[tbooth]

Hi,

FWIW, I seem to have made duc index hang by inspecting a directory containing 8.1 million files. On the other hand, now I realised that this directory exists on my system I may well have found out where all my disk space went to!

TIM

[zevv]

Quoting Tim Booth (2020-03-18 17:42:26)

FWIW, I seem to have made duc index hang by inspecting a directory containing 8.1 million files.

Ha! So it was not hanging but just chomping away. Good that you found the issue!

-- :wq ^X^Cy^K^X^C^C^C^C

[l8gravely]

"Ico" == Ico Doornekamp [email protected] writes:

Ico> Quoting Tim Booth (2020-03-18 17:42:26)

FWIW, I seem to have made duc index hang by inspecting a directory containing 8.1 million files.

Ico> Ha! So it was not hanging but just chomping away. Good that you found Ico> the issue!

Heh, I just ran into this issue too at $WORK, where a server had probably a million small files in /tmp, which really made it slow and unhappy when you tried to do an 'ls' in there.

The old Usenet news servers had the same problems, and they ended up making multiple levels of directories and hashing files into multiple levels to get around this issue, since Unix/Linux filesystems don't scale well will that many files in a directory.

Hmm... this might be a good feature, where we add in a total file count per-directory to duc, and then figure out how to show that number nicely. Not the file count for a directory and all sub-directories, but just a per-directory count.

John