SOLR-16187: ExecutorUtil#awaitTermination shouldn't wait forever

[risdenk]

https://issues.apache.org/jira/browse/SOLR-16187

[madrob]

Relevant ideas from Mark's pass at this last year - https://github.com/markrmiller/solr/blob/reference_impl/solr/solrj/src/java/org/apache/solr/common/util/ExecutorUtil.java#L71-L97

I think there's actually a bug in his code regarding interrupt, but I do like throwing an exception if we fail to shut down. This is the sort of thing that should probably fail loudly instead of going away quietly?

[risdenk]

@madrob agreed would be nice to throw an exception potentially. Thanks for the reference as well looks plausible and shortens the shutdown time if things behave.

[risdenk]

So fun fact - https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/ExecutorService.html has the same example of this:

void shutdownAndAwaitTermination(ExecutorService pool) {
   pool.shutdown(); // Disable new tasks from being submitted
   try {
     // Wait a while for existing tasks to terminate
     if (!pool.awaitTermination(60, TimeUnit.SECONDS)) {
       pool.shutdownNow(); // Cancel currently executing tasks
       // Wait a while for tasks to respond to being cancelled
       if (!pool.awaitTermination(60, TimeUnit.SECONDS))
           System.err.println("Pool did not terminate");
     }
   } catch (InterruptedException ex) {
     // (Re-)Cancel if current thread also interrupted
     pool.shutdownNow();
     // Preserve interrupt status
     Thread.currentThread().interrupt();
   }
 }

[risdenk]

@madrob / @magibney I updated this PR to latest main and added throwing an exception if the pool couldn't be shutdown.