sgx-lkl icon indicating copy to clipboard operation
sgx-lkl copied to clipboard

Integrate virtio channel signalling with lthread scheduler

Open prp opened this issue 4 years ago • 2 comments

Currently, lthreads indicate that they are sleeping on an event channel directly by setting the waiting flag in the event channel. This causes the host side to signal condition variables that may have nothing sleeping on them, because the lthread scheduler still has other work to do.

The correct way of doing this would be for an lthread to specify to the scheduler when it yields which event channels it is waiting for. The scheduler would then keep these in a separate queue for each event channel. When there are no non-sleeping threads available, the scheduler would set the waiters bit and ocall (ideally with the list of event channels and the list of expected values).

It is possible that we're also missing wakeups because we are not checking the event channels when we hold the mutex on the outside, so we can:

  1. Mark an event channel as having waiters in the enclave.
  2. Ocall
  3. From the device thread, see that there are waiters and signal the condition variable.
  4. From the ocall, acquire the mutex and sleep on the condition variable, missing the wakeup.

This won't matter very much, because we also have a timeout for timers, so we'll get a timer wakeup, but if we didn't have a timer wakeup scheduled then we'd sleep indefinitely. To fix this, we need to modify the sleep call to take an array of event channels that it is sleeping on, so the sequence is:

  1. Mark the event channel as having waiters in the enclave and record its value.
  2. Ocall with the event channel and the last-read value of the event channel.
  3. From the device thread, see that there are waiters and signal the condition variable.
  4. From the ocall
  5. Acquire the mutex
  6. Check that the event channels have the same values as expected.
  7. See that they do not, unlock the mutex, and return immediately.

prp avatar May 05 '20 14:05 prp

I'm picking up work on this.

SeanTAllen avatar Jun 24 '20 17:06 SeanTAllen

At this point, @prp believes this is an efficiency but not correctness problem.

SeanTAllen avatar Jul 29 '20 16:07 SeanTAllen