Fix known issues with multithreaded mode

[mpartel]

The man page documents some known caveats with --multithreaded. They should be fixed so we can confidently default to multithreaded mode. A simple global lock around accessing file attributes should solve the problem, but this needs to be thought through carefully.

Lock forwarding (#36) needs multithreaded mode.

[gdoffe]

Hi @mpartel I am facing some kind of deadlock with --multithreaded and --enable-lock-forwarding. I am using bindfs to share between users download and build caches of Yocto project builds. There are heavy accesses to these shared directories. If I launch two builds in parallel, both builds get stuck infinitely. bindfs is mounted on a ext4 fs with the following command line: /usr/local/bin/bindfs --enable-lock-forwarding --multithreaded --mirror-only=@docker --create-for-user=${DIRECTORY_OWNER} --create-for-group=${DIRECTORY_GROUP} ${DIRECTORY} ${DIRECTORY}

The version I use is 1.16.1

My questions are the following:

• The caveats you mention in this quite old issue, are they solved in 1.16.1 ?
• Do you see any mistake in my command line that could explain the behavior I mention ?
• Debug output does not show any error, what are your recommandations to investigate further ?

:pray:

[mpartel]

The caveats mentioned have not been fixed unfortunately, but that wouldn't help with this specific problem.

I suspect that libfuse's multithreaded mode uses a fixed-size thread pool. That would mean that heavy usage can still cause a deadlock: if there are N threads and all are busy waiting for some lock, then bindfs can receive no more requests, including requests to unlock. I don't see a way around this in libfuse's API.

Since you're mounting the directory on itself, are you sure you need --enable-lock-forwarding? If all access goes through bindfs, then there should be no need to forward locks to the underlying FS.

[gdoffe]

Thanks for your quick reply.

Since you're mounting the directory on itself, are you sure you need --enable-lock-forwarding? If all access goes through bindfs, then there should be no need to forward locks to the underlying FS.

Yes all accesses to the directory are made only with bindfs. So your comment is really interesting. I added --enable-lock-forwarding option because otherwise I face some "permission denied" on some files. I supposed (from very far :wink: ) that it could be related to the fact that the file was locked but as the lock is not forwarded it leads to a permission denied. (But I admit it is an ignorant guess :wink: ).

For example:

ERROR: sqlite3-native-3_3.38.2-r0 do_configure: Build of do_configure failed
ERROR: sqlite3-native-3_3.38.2-r0 do_configure: Traceback (most recent call last):
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/build.py", line 763, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/build.py", line 737, in _exec_task
    event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/test/yocto-build/layers/poky/meta/classes/sstate.bbclass", line 1188, in defaultsstate_eventhandler
    bb.siggen.dump_this_task(siginfo, d)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/siggen.py", line 727, in dump_this_task
    bb.parse.siggen.dump_sigtask(fn, task, outfile, "customfile:" + referencestamp)
  File "/home/test/yocto-build/layers/poky/meta/lib/oe/sstatesig.py", line 214, in dump_sigtask
    super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/siggen.py", line 375, in dump_sigtask
    bb.utils.mkdirhier(os.path.dirname(sigfile))
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/utils.py", line 740, in mkdirhier
    raise e
  File "/home/test/yocto-build/layers/poky/bitbake/lib/bb/utils.py", line 737, in mkdirhier
    os.makedirs(directory)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/mnt/sstate/universal/f8'

ERROR: Task (virtual:native:/home/test/yocto-build/layers/poky/meta/recipes-support/sqlite/sqlite3_3.38.2.bb:do_configure) failed with exit code '1'

However I forgot an important point: I am building inside a Docker container, doing bind mount of sub-directories of the bindfs mount. If I do not use docker to build, I do not reproduce the permission denied problem...

[mpartel]

Ok, that might well be caused by the race conditions that this issue is about. So something like this:

1. process A creates a file but bindfs hasn't yet executed the --create-for-* flags on it.
2. process B finds that the file exists and tries to access it, but the permissions aren't correct yet.

No idea why only Docker triggers it though.