[BUG] MLResourceNotFoundException leads to 500 error

[dbwiddis]

What is the bug?

The MLResourceNotFoundException is just a decorated RuntimeException with special handling for ML Stats.

However, it's used in cases where a document is not found which should generally result in a 404 error.

How can one reproduce the bug? Steps to reproduce the behavior:

• Follow any code path leading to the MLResourceNotFound exception handling in ModelAccessControlHelper.validateModelGropuAccess() which would be trying to register a model to a model group that doesn't exist, or having the model group index not exist. This may be difficult to encounter the exception due to being caught earlier in the handling, but is appearing in the MultiTenancy branch due to tenant ID checks.

{
  "error": {
    "root_cause": [
      {
        "type": "m_l_resource_not_found_exception",
        "reason": "Fail to find model group"
      }
    ],
    "type": "m_l_resource_not_found_exception",
    "reason": "Fail to find model group"
  },
  "status": 500
}

What is the expected behavior?

A resource "not found" should result in a 4xx series error, usually a 404.

Do you have any additional context?

An OpenSearchStatusException would probably be appropriate here but it's not quite as simple since we have to keep the current stats exclusion exceptions, so a more careful refactoring is needed.