Race condition between table creation and namespace deletion

[keith-turner]

Describe the bug

There may be a race condition where a namesapce could be deleted that has tables. The check for tables in a namespaces was only observed in code on the client side. After this client code does the check it starts a fate operation to delete the namespace. After the client side check is done, not seeing any other checks in the code. If there are no other checks then a table could be created in the namespace after that client side check is done. This would result in a table w/o a namespace.

Expected behavior

A good way to handle this would be to do check for tables after exclusively acquiring the namespace lock in the fate operation to delete the namespace. If tables are found then an exception should be propagated back to the client. The client side check should be removed in order to make testing the server side code possible.

The correctness of this solution requires that all fate table operations get a read lock on the namespace, which is currently done.

[keith-turner]

If any changes are made for this in a bug fix release like 2.1.4 it would be tricky to remove the client side check. That could lead to a situation where neither client or server are doing the check if using a 2.1.4 client and 2.1.3 server.

[ctubbsii]

@meatballspaghetti this issue might be somewhat adjacent to the table name mapping code in the main branch that you're working on for #4698 , because some table map updates might make assumptions about the namespace mapping current state. I'm not sure if the race condition is really possible or not, but it's something to watch for when updating the mapping code for #4698

[ctubbsii]

This would almost certainly be a non-issue with #5414

[keith-turner]

Work on this would be best done after #4698 is done. See comments on #5443 for more details.