[Core][2/n] Core structured logging: add Text formatter and pass log config to worker process

[kevin85421]

trafficstars

Why are these changes needed?

• Add a TextFormatter. It is a formatter that converts LogRecord into a human-readable message string.

• Add a predefined logging configuration. This allows users to easily configure Ray task and actor loggers to use TextFormatter, and to add arbitrary attributes to log messages.

• Example

  import ray
  import sys
  import logging
  from ray._private.structured_logging.utils import LoggingConfig
  
  ray.init(
      job_config=ray.job_config.JobConfig(py_log_config=LoggingConfig("TEXT"))
  )
  
  def init_logger():
      return logging.getLogger()
  
  logger = logging.getLogger("ray")
  logger.info("Driver process", extra={"username": "johndoe"})
  
  @ray.remote
  def f():
      logger = init_logger()
      logger.info("A Ray task")
  
  @ray.remote
  class actor:
      def __init__(self):
          pass
      def print_message(self):
          logger = init_logger()
          logger.info("A Ray actor")
  
  task_obj_ref = f.remote()
  ray.get(task_obj_ref)
  
  actor_instance = actor.remote()
  ray.get(actor_instance.print_message.remote())
  

• Output

  2024-05-21 06:48:06,699 INFO worker.py:1740 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8266
  2024-05-21 06:48:07,333 INFO test_logging.py:14 -- Driver process
  (f pid=3772046) 2024-05-21 06:48:07,349 INFO test_logging.py:19 -- A Ray task job_id=01000000 worker_id=87cac8ad9f50f185f97d0fc36e05616669c3e00a135bc0f443a709ac node_id=683c682fee5d431f3f4cf25fa992df9a7918b1c32e88943e853ad06b task_id=c8ef45ccd0112571ffffffffffffffffffffffff01000000
  (actor pid=3773875) 2024-05-21 06:48:07,870     INFO test_logging.py:27 -- A Ray actor job_id=01000000 worker_id=b362329f5dab902df9e3fc0fe73df6279c661baaa709854e135378aa node_id=683c682fee5d431f3f4cf25fa992df9a7918b1c32e88943e853ad06b actor_id=c4fc669d536be18637fa490201000000 task_id=c2668a65bda616c1c4fc669d536be18637fa490201000000
  

Related issue number

Checks

• [ ] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• [ ] I've run scripts/format.sh to lint the changes in this PR.
• [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
  • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
• [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • [ ] Unit tests
  • [ ] Release tests
  • [ ] This PR is not tested :(

[jjyao]

@rkooo567 can you also check the API to see if it makes sense?

[kevin85421]

• Case 1: pb.serialized_py_logging_config is 428 bytes.

  log_dict_config = {
      "version": 1,
      "disable_existing_loggers": False,
      "formatters": {
          "text": {
              "()": "ray._private.structured_logging.formatters.TextFormatter",
          },
      },
      "filters": {
          "core_context": {
              "()": "ray._private.structured_logging.filters.CoreContextFilter",
          },
      },
      "handlers": {
          "console": {
              "level": "INFO",
              "class": "logging.StreamHandler",
              "formatter": "text",
              "filters": ["core_context"],
          },
      },
      "root": {
          "level": "INFO",
          "handlers": ["console"],
      },
  }
  ray.init(
       job_config=ray.job_config.JobConfig(py_logging_config=LoggingConfig(dict_config))
  )
  

• Case 2: pb.serialized_py_logging_config is 95 bytes.

  ray.init(
       job_config=ray.job_config.JobConfig(py_logging_config=LoggingConfig("TEXT"))
  )