resource leak in ray/pthon/ray/node.py

[09wakharet]

What is the problem?

The script below returns ResourceWarning: Enable tracemalloc to get the object allocation traceback many times. It seems to be a resource leak in ray/pthon/ray/node.py, due to unclosed logging files. The same problem can also be seen by running ray/rllib/utils/exploration/tests/test_explorations.py as-is.

Reproduction (REQUIRED)

import ray
import sys
import unittest

class TestResourceLeak(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        ray.init(local_mode=True)

    @classmethod
    def tearDownClass(cls):
        ray.shutdown()

    def test_useless(self):
        print("RAN")

if __name__ == "__main__":
    import pytest
    sys.exit(pytest.main(["-v", __file__]))

This will give the stack trace:

WARNING:tensorflow:From /opt/miniconda3/envs/ray/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2020-07-17 13:11:41,465	INFO resource_spec.py:223 -- Starting Ray with 4.44 GiB memory available for workers and up to 2.23 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis-shard_0.err' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis-shard_0.out' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis.err' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis.out' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:740: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/gcs_server.out' mode='a' encoding='utf-8'>
  self.start_gcs_server()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:740: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/gcs_server.err' mode='a' encoding='utf-8'>
  self.start_gcs_server()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:742: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/monitor.out' mode='a' encoding='utf-8'>
  self.start_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:742: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/monitor.err' mode='a' encoding='utf-8'>
  self.start_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2020-07-17 13:11:41,918	INFO services.py:1193 -- View the Ray dashboard at localhost:8265
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:747: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/dashboard.out' mode='a' encoding='utf-8'>
  self.start_dashboard(require_dashboard=False)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:747: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/dashboard.err' mode='a' encoding='utf-8'>
  self.start_dashboard(require_dashboard=False)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:755: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/plasma_store.out' mode='a' encoding='utf-8'>
  self.start_plasma_store()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:755: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/plasma_store.err' mode='a' encoding='utf-8'>
  self.start_plasma_store()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:756: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/raylet.out' mode='a' encoding='utf-8'>
  self.start_raylet()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:756: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/raylet.err' mode='a' encoding='utf-8'>
  self.start_raylet()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:757: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/reporter.out' mode='a' encoding='utf-8'>
  self.start_reporter()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:757: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/reporter.err' mode='a' encoding='utf-8'>
  self.start_reporter()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:760: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/log_monitor.out' mode='a' encoding='utf-8'>
  self.start_log_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:760: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/log_monitor.err' mode='a' encoding='utf-8'>
  self.start_log_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 79521 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 79524 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:978: ResourceWarning: unclosed file <_io.BufferedWriter name=8>
  allow_graceful=allow_graceful)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

If we cannot run your script, we cannot fix your issue.

• [ ] I have verified my script runs in a clean environment and reproduces the issue.
• [ ] I have verified the issue also occurs with the latest wheels.

[atomr3]

Great find

[chunyang-wen]

Same problem met. Just call ray.init() in one of the test functions, will get this warning.

[miguel-angel-monjas]

Same comment with pretty similar code:

class RayTestMethods(unittest.TestCase):

    def setUp(self):
        self.ray_client = RayClient()

    def test_model_addition(self):
        pass

    def tearDown(self):
        self.ray_client.shutdown()

[rkooo567]

cc @edoakes Wasn't this intentional? What's the best way to remove warning here?

[rkooo567]

Couldn't reproduce the issue in the master (see output below). Closing it! Please reopen if you still see the issue.

Test session starts (platform: darwin, Python 3.8.13, pytest 5.4.3, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/sangbincho/work/ray
plugins: asyncio-0.16.0, anyio-3.6.1, sugar-0.9.4, timeout-2.1.0, shutil-1.7.0, rerunfailures-10.2, virtualenv-1.7.0, repeat-0.9.1, lazy-fixture-0.6.3
collecting ... 
 a.py::TestResourceLeak.test_useless ✓                                                                                               100% ██████████
================================================================= warnings summary =================================================================
a.py::TestResourceLeak::test_useless
  /Users/sangbincho/work/ray/python/ray/_private/client_mode_hook.py:105: DeprecationWarning: DeprecationWarning: local mode is an experimental feature that is no longer maintained and will be removed in the future.For debugging consider using Ray debugger. 
    return func(*args, **kwargs)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Results (1.50s):
       1 passed
(core) sangbincho@SangBins-MacBook-Pro-16-inch-2019 ray % 

[zhoubin-me]

use pytest instead:

pytest pytest/test_unittest_xxx.py

[dustinrb]

Unfortunately this is still an issue.

Running a test job with unittest (python -m unittest test_ray.py) produces multiple ResourceWarnings like the original comment by @09wakharet.

# contents of test_ray.py
import ray
import unittest

class RayTest(unittest.TestCase):
    def setUp(self):
        self.ray_ctx = ray.init(num_cpus=1, num_gpus=0)

    def tearDown(self):
        ray.shutdown()

    def test_method(self):
        pass

I have tested this with Python 3.10 on macOS Big Sur/Ray==2.8 and macOS Sonoma/Ray==2.9.

[samarth-contextual]

Doesn't look like this got resolved.

python 3.10 // ubunutu 22.04 // CUDA 12.2

[HenryL27]

I ran into this when integrating with the weaviate client:

import weaviate
import ray
ray.init()

python 3.11 // macos 14.4.1

[hsm207]

I also ran into this when integration with the weaviate client.

Python 3.11.8 // Debian GNU/Linux 12 (bookworm) // ray 2.10.0

[hsm207]

update:

I don't think this is actually related to the weaviate client.

I wrote a script named bug.py that has just this:

import ray
ray.init()

and ran it with:

python -Wd bug.py

and the warnings appear too.

Could the problem be that the log file streams created in open_log are not closed elsewhere?