Recursive RW locks

[mdorier]

It would be very useful to provide recursive RW lock.

It is not uncommon for a code to read-lock a structure and eventually write-lock it for a small modification. Right now we can either unlock it then write-lock it, which is incorrect because it gives a chance for other threads to modify it so the structure will be inconsistent with what was read by the thread that now wants to modify it; or we can use a mutex, which basically write-lock the structure completely for a longer time than actually needed.

[halimamer]

We can extend the RW lock interface to take attributes similar to the mutex interface. No ETA for this feature. If you are on a hurry, my advice is to encapsulate a RW lock inside a structure and track ownership and nesting levels similar to what we do for mutexes (https://github.com/pmodels/argobots/blob/master/src/mutex.c#L152)