aptly db cleanup hangs at end?

[dankegel]

Just now, I saw a run of aptly db cleanup take 11 minutes. ps showed it using very little cpu time. I strace'd it starting at 10 minutes; here's what I saw: futex(0xe3a6d8, FUTEX_WAIT, 0, NULL) = ? it just sat there for about a minute, then: PANIC: attached pid 29401 exited with 0

So evidently aptly db cleanup finished a while ago, but was held up by some thread that was sitting on a futex for a long time?

It's bad to monopolize the apt server for ten minutes, it holds up many builds.
I've only seen this happen once, but I'll instrument our script to see how long aptly operations take.

I have been running aptly db cleanup after each upload, which is overkill; maybe I'll dial that back if it's going to misbehave like this.

[dankegel]

I've seen it several times recently. This is with aptly version: 0.9.5, guess I should try updating.

Just now, after 15 minutes of wall time, I tried attaching to it with gdb. It's still doing disk I/O:

#1  0x00000000005d84ce in syscall.Getdents (fd=7, buf=..., n=100, err=...) at /home/smira/.gvm/gos/go1.4/src/syscall/zsyscall_linux_amd64.go:519
#2  0x00000000005d65b8 in syscall.ReadDirent (fd=7, buf=..., n=0, err=...) at /home/smira/.gvm/gos/go1.4/src/syscall/syscall_linux.go:738
...

Most of the time strace shows lots of those futex lines, but occasionally some lstat calls. So possibly what's happening is the box is just overloaded by too many things doing disk I/O. And, fun fact, the directory it's frobbing is on a slow disk.

So, this is probably expected.

[smira]

aptly db cleanup internally does full listing of files in the aptly pool trying to figure out if some package files are not used anymore.

[flotho]

hi @dankegel could you retry with recent code please