Frank Filz

Could you add back trace of all Ganesha threads?

Sorry I missed those back traces. Can you run with full debug info packages installed so we get line numbers in the back traces?

There is this fix in V6-dev.3 that might help: 764015ddf9dc6b40568d6abe20b8fec239e25222 Decrement fd_work while reclaiming if FD is busy Also maybe this one: https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1177971?usp=search

I think the next step is to turn on FSAL debugging: LOG { COMPONENTS { FSAL = FULL_DEBUG; } } I suspect we still have a leaked io_work or fd_work...

Hmm, this patch may also have relevance, though it is only for FSAL_GPFS and needs some consideration and maybe generalization: https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1184121

Yea, lock inversion deadlock is what I'm thinking seeing threads waiting to acquire two different locks.

This could also be https://github.com/nfs-ganesha/nfs-ganesha/issues/1102 You could enable debug on COMPONENT_RW_LOCK: In config: ``` LOG { COMPONENTS { RW_LOCK = FULL_DEBUG; } } ```

Thanks for the verify. And until you typed it, I didn't notice the change number :-)

BTW, review comments would be appreciated in Gerrithub. Also, feel free to do a Verify +1.

Yea we probably should do that. We need to make sure the cached attributes also match what was successfully changed.