Setting log level in RayDP

[gbraes]

In the recent version of Ray/RayDP 1.12/0.4.2 I am getting a lot of INFO logs. I am trying to disable them. Does Anyone know how to do that when using RayDP? sc.sparkContext.setLogLevel('WARN') does not work.

[gbraes]

I have also tried to disable it using logging_level='warn' in Ray.init

import ray
import raydp
from pyspark import SparkContext, SparkConf, SQLContext


configs = {}
ray.init(logging_level='warn') 
sc = raydp.init_spark(app_name='RayDP stream Example', num_executors=2, executor_cores=2, executor_memory='4GB',configs=configs)

It keeps giving you a lot of INFO logs.

Ray: 1.12 RayDP: 0.4.2 Pyspark: 3.2.1 Java version: Java 11 (build 11.0.15+10-Ubuntu-0ubuntu0.20.02.1)

[kira-lin]

Hi, It seems like only under ray.init('auto', logging_level='warn') and add configs={"spark.driver.extraJavaOptions": "-Dlog4j.configuration=/home/lzhi/log4j.properties"} in raydp.init_spark can the logging level be set. We'll look into it.

Thanks for raising this issue

[kira-lin]

Hi @gbraes, This problem is ray's jar wraps slf4j, while pyspark/jars/ also contains slf4j. When spark starts, we need to add ray.jar to classpath, so it'll find two bindings, and this is annoying. As I tested, if you remove the slf4j jar from pyspark/jars/, and set the configs={"spark.driver.extraJavaOptions": "-Dlog4j.configuration=/home/lzhi/log4j.properties"}, most logging will go away.

[gbraes]

Hi @kira-lin , I have tested using my own log4j.properties file where I set up the log level to warn. After adding that extra java option where I point to that file, as you mentioned in the config when loading raydp, and now it works!. So, now I don't get the annoying and verbose INFO logs. Thanks!

However, I did not need to remove slf4j jar from pyspark.

[kira-lin]

That is also possible, because as slf4j says, it's somehow random(up to jvm) to decide which one to load when multiple jars is found in classpath. Glad to hear that you are able to turn it off.

[carsonwang]

This has been improved in RayDP 0.5.0. Please see https://github.com/oap-project/raydp/blob/master/doc/spark_on_ray.md#logging

[gebraes]

This has been improved in RayDP 0.5.0. Please see https://github.com/oap-project/raydp/blob/master/doc/spark_on_ray.md#logging

@carsonwang Thanks a lot!