Spark job Running forever after completed

Open sontt710 opened this issue 1 year ago • 3 comments

Hello,

I try to use appender with the following version to submit spark job (spark-pi example) on k8s, but the job running forever and not shutdown after complete, can you help me to check

Version

Spark 3.3.4 Log4j2 Appender 5.3.2 hadoop-aws-3.3.2.jar aws-java-sdk-bundle-1.11.1026.jar aws-java-sdk-s3-1.12.772.jar

Driver log

Oct 07 '24 08:10 sontt710

If you are quitting a process, see this to shut down: https://github.com/bluedenim/log4j-s3-search-samples/blob/master/appender-log4j2-sample/src/main/java/com/van/example/Main.java#L43

Oct 07 '24 08:10 bluedenim

Thanks @bluedenim, I understand, I will have to shut it down myself in the application, there's no way to configure it from the log4j2 XML file?

Oct 07 '24 10:10 sontt710

There is no easy way for the event log thread to know when your program should end. Only your program knows when it's time to end.

Oct 12 '24 00:10 bluedenim