modin Reading large json lines dataset fails because of rows having different columns

Reading large json lines dataset fails because of rows having different columns

Open WaDelma opened this issue 2 years ago • 2 comments

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.4 LTS
Modin version (modin.__version__): 0.15.3
Python version: 3.10.6
Code we can use to reproduce:

import modin.pandas as pd
pd.read_json("Automotive_5.json", lines=True)

where Automotive_5.json is the "small" (1711519 lines) automotive dataset from https://nijianmo.github.io/amazon/index.html I tried to reproduce it by generating synthetic json lines file where different json objects have different fields with no avail. Have not yet tried to reduce the dataset to smaller to see when it stops failing.

Describe the problem

I am trying to load large json lines file (which is actually the whole dataset, but it reproduces with the smaller one) and it errors out with ValueError('Columns must be the same across all rows.'). I can work around this by loading the dataset with pandas and then transforming that to moding dataframe, but that is unnecessarily slow.

Source code / logs

output.txt

Sep 10 '22 17:09 WaDelma

@WaDelma thank you for opening this issue! I am able to reproduce this on the latest master.

@mvashishtha I took a look and it seems like we're making an incorrect assumption about the columns of the resulting dataframe from the first line of the JSON file. For example, in json_dispatcher, we try to get a gist of all the possible columns by taking a look at the first line of the JSON file. However, it is possible that other lines will have key-value pairs that do not exist in the first line of the file. Does this fix fall under the things you've been working on already?

Sep 12 '22 13:09 pyrito

@pyrito yep, each line in the file input to read_json with lines=True can have any columns. e.g. reading this file with read_json(path, lines=False) works in pandas but Modin gives the error in this issue's original post.

{"a": 1}
{"b": 2}

The JSON parsing work I've been doing is not related to this.

There are a few ways in which the current read_json doesn't work when different lines have different columns:

we always assume that different row partitions of a modin frame have the same columns.
read_json uses the number of columns in the first line to determine column partition widths (through json_dispatcher calling _define_metadata). this will still work with mixed columns but reading the first row might not tell us how many columns we really need to split
we use the columns from the first line as the columns of the entire modin dataframe. That's another bug.

We will have to rewrite read_json so it does something like

read row partitions
do some kind of copartition between the row partitions.
create a modin frame out of the copartitioned row partitions.

Maybe we could parse modin frames with just 1 column partition each, modin.pandas.concat these, and then partition the result column wise?

@WaDelma unfortunately I don't think anyone will be able to take this issue on at the moment, but we hope to address it soon. Meanwhile, I hope you can read your json into a pandas dataframe with pandas.read_json, then convert it to a Modin dataframe with modin.pandas.DataFrame(pandas_dataframe), then use Modin.

Sep 13 '22 21:09 mvashishtha

Hey, @mvashishtha

Did you solve the bugs for read_json command?

I have 14gigs of Json file -1 file- and having an error when trying to import as a dataframe

I'm getting runtime, and key error for engine :/

Dec 10 '22 17:12 mertbozkir

@mertbozkir this bug is still open and hasn't been fixed. What errors exactly are you getting? Are you able to provide a stack trace for both errors?

Dec 10 '22 17:12 mvashishtha

It's long error, I couldn't copied the rest :/

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 532, in start
    raise RuntimeError(f"{type(self).__name__} failed to start.") from exc
RuntimeError: Nanny failed to start.
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 513, in start
    raise self.__startup_exc
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 524, in start
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 31, in initialize_dask
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    client = default_client()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 5202, in default_client
    return await fut
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 366, in start_unsafe
    response = await self.instantiate()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 707, in start
    await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/process.py", line 38, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/process.py", line 201, in _start
    process.start()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
    raise ValueError(
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
ValueError: No clients found
Start a client and point it to the scheduler address
  from distributed import Client
  client = Client('ip-addr-of-scheduler:8786')


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    _check_not_importing_main()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    df = pd.read_json("matches.json")
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/pandas/__init__.py", line 161, in _update_engine
    initialize_dask()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 47, in initialize_dask
    client = Client(n_workers=num_cpus, memory_limit=worker_memory_limit)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 988, in __init__
    self.start(timeout=timeout)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 1185, in start
    sync(self.loop, self._start, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 406, in sync
    raise exc.with_traceback(tb)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 379, in f
    result = yield future
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 1251, in _start
    self.cluster = await LocalCluster(
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/spec.py", line 400, in _
    await self._correct_state()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/spec.py", line 369, in _correct_state_internal
    await w  # for tornado gen.coroutine support
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 513, in start
    raise self.__startup_exc
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 524, in start
    await asyncio.wait_for(self.start_unsafe(), timeout=timeout)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 366, in start_unsafe
    response = await self.instantiate()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 707, in start
    await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/process.py", line 38, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/process.py", line 201, in _start
    process.start()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
2022-12-10 20:23:57,630 - distributed.nanny - WARNING - Restarting worker
2022-12-10 20:23:57,644 - distributed.nanny - WARNING - Restarting worker

Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 848, in _wait_until_connected
    msg = self.init_result_q.get_nowait()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/queues.py", line 135, in get_nowait
    return self.get(False)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/queues.py", line 116, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 545, in _on_worker_exit
    await self.instantiate()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 714, in start
    msg = await self._wait_until_connected(uid)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 850, in _wait_until_connected
    await asyncio.sleep(self._init_msg_interval)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    return await future
asyncio.exceptions.CancelledError

Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 848, in _wait_until_connected
    msg = self.init_result_q.get_nowait()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/queues.py", line 135, in get_nowait
    return self.get(False)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/queues.py", line 116, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 545, in _on_worker_exit
    await self.instantiate()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 442, in instantiate
    result = await self.process.start()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 714, in start
    msg = await self._wait_until_connected(uid)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/nanny.py", line 850, in _wait_until_connected
    await asyncio.sleep(self._init_msg_interval)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    return await future
asyncio.exceptions.CancelledError
2022-12-10 20:23:57,735 - distributed.nanny - WARNING - Restarting worker
2022-12-10 20:23:57,735 - distributed.nanny - WARNING - Restarting worker
2022-12-10 20:23:57,739 - distributed.nanny - WARNING - Restarting worker
UserWarning: Dask execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    from distributed import Client

    client = Client()

UserWarning: Dask execution environment not yet initialized. Initializing...
To remove this warning, run the following python code before doing dataframe operations:

    from distributed import Client

    client = Client()

^CTraceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 260, in get
Traceback (most recent call last):
    raw = cls._get_raw_from_config()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 51, in _get_raw_from_config
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 260, in get
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 260, in get
    raw = cls._get_raw_from_config()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 51, in _get_raw_from_config
    raw = cls._get_raw_from_config()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 51, in _get_raw_from_config
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 31, in initialize_dask
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 31, in initialize_dask
    return os.environ[cls.varname]
    return os.environ[cls.varname]
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/os.py", line 679, in __getitem__
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/os.py", line 679, in __getitem__
    client = default_client()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 5202, in default_client
    client = default_client()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 5202, in default_client
    return os.environ[cls.varname]
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/os.py", line 679, in __getitem__
    raise KeyError(key) from None
    raise KeyError(key) from None
KeyError: 'MODIN_ENGINE'
KeyError: 'MODIN_ENGINE'

During handling of the above exception, another exception occurred:


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
    main_content = runpy.run_path(main_path,
    raise KeyError(key) from None
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
KeyError: 'MODIN_ENGINE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    raise ValueError(
ValueError: No clients found
Start a client and point it to the scheduler address
  from distributed import Client
  client = Client('ip-addr-of-scheduler:8786')


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    return _run_module_code(code, init_globals, run_name,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    raise ValueError(
    df = pd.read_json("matches.json")
ValueError: No clients found
Start a client and point it to the scheduler address
  from distributed import Client
  client = Client('ip-addr-of-scheduler:8786')
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    df = pd.read_json("matches.json")
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    return obj(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
    return _run_module_code(code, init_globals, run_name,
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
  File "/Users/mertbozkir/Desktop/Code/lol/venv_l    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/pandas/__init__.py", line 136, in _update_engine
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
    publisher.get() == "Native"
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 262, in get
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    return _run_module_code(code, init_globals, run_name,
    cls._value = cls._get_default()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 111, in _get_default
ol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    import distributed
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/__init__.py", line 42, in <module>
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    df = pd.read_json("matches.json")
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    df = pd.read_json("matches.json")
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
    from distributed.diagnostics.progressbar import progress
  File "/Users/mertbozkir/Desktop/Code/lol/ven    return obj(*args, **kwargs)
    return obj(*args, **kwargs)
v_lol/lib/python3.10/site-packages/distributed/diagnostics/progressbar.py", line 9, in <module>
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/pandas/__init__.py", line 136, in _update_engine
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    publisher.get() == "Native"
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 262, in get
    Engine.subscribe(_update_engine)
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/    from timeit import default_timer
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
    cls._value = cls._get_default()
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 111, in _get_default
  File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1411, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1577, in find_spec
  File "<frozen importlib._bootstrap_external>", line 161, in _path_isfile
  File "<frozen importlib._bootstrap_external>", line 153, in _path_is_mode_type
  File "<frozen importlib._bootstrap_external>", line 147, in _path_stat
KeyboardInterrupt
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/pandas/__init__.py", line 136, in _update_engine
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/pytho    import distributed
n3.10/site-packages/modin/pandas/__init__.py", line 161, in _update_engine
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/__init__.py", line 15, in <module>
    from distributed.actor import Actor, ActorFuture, BaseActorFuture
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/actor.py", line 14, in <module>
    publisher.get() == "Native"
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 262, in get
    initialize_dask()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 47, in initialize_dask
    cls._value = cls._get_default()
    client = Client(n_workers=num_cpus, memory_limit=worker_memory_limit)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/envvars.py", line 111, in _get_default
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 988, in __init__
    from distributed.client import Future
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 57, in <module>
    import distributed
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/__init__.py", line 15, in <module>
    from distributed import cluster_dump, preloading
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/preloading.py", line 18, in <module>
    from distributed.actor import Actor, ActorFuture, BaseActorFuture
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/actor.py", line 14, in <module>
    self.start(timeout=timeout)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 1185, in start
    from distributed.core import Server
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 28, in <module>
    from distributed.client import Future
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 57, in <module>
    from distributed.comm import (
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/__init__.py", line 48, in <module>
    from distributed import cluster_dump, preloading
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/preloading.py", line 18, in <module>
    return _run_module_code(code, init_globals, run_name,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
    from distributed.core import Server
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/core.py", line 28, in <module>
    _register_transports()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/__init__.py", line 22, in _register_transports
    from distributed.comm import inproc, ws
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/ws.py", line 12, in <module>
    _run_code(code, mod_globals, init_globals,
    from distributed.comm import (
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    sync(self.loop, self._start, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/__init__.py", line 48, in <module>
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 402, in sync
    wait(10)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 391, in wait
    _register_transports()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/__init__.py", line 43, in _register_transports
    from tornado import web
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/tornado/web.py", line 71, in <module>
    return e.wait(timeout)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/threading.py", line 607, in wait
    from distributed.comm import ucx
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/comm/ucx.py", line 31, in <module>
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    import http.cookies
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/http/cookies.py", line 187, in <module>
    signaled = self._cond.wait(timeout)
    from distributed.diagnostics.nvml import (
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/diagnostics/nvml.py", line 13, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/threading.py", line 324, in wait
    import pynvml
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/pynvml.py", line 33, in <module>
    df = pd.read_json("matches.json")
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    _OctalPatt = re.compile(r"\\[0-3][0-7][0-7]")
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/re.py", line 251, in compile
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
    from ctypes.util import find_library
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/ctypes/util.py", line 71, in <module>
    from ctypes.macholib.dyld import dyld_find as _dyld_find
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/ctypes/macholib/dyld.py", line 7, in <module>
    from ctypes.macholib.dylib import dylib_info
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    return _compile(pattern, flags)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/re.py", line 303, in _compile
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 975, in get_code
  File "<frozen importlib._bootstrap_external>", line 1074, in get_data
KeyboardInterrupt
    p = sre_compile.compile(pattern, flags)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/sre_compile.py", line 792, in compile
    code = _code(p, flags)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/sre_compile.py", line 628, in _code
    return obj(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    _compile_info(code, p, flags)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/sre_compile.py", line 564, in _compile_info
    lo, hi = pattern.getwidth()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/sre_parse.py", line 204, in getwidth
    hi = hi + 1
KeyboardInterrupt
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 31, in initialize_dask
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/pandas/__init__.py", line 161, in _update_engine
    initialize_dask()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 47, in initialize_dask
    client = Client(n_workers=num_cpus, memory_limit=worker_memory_limit)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 988, in __init__
    self.start(timeout=timeout)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 1185, in start
    client = default_client()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 5202, in default_client
    raise ValueError(
ValueError: No clients found
Start a client and point it to the scheduler address
  from distributed import Client
  client = Client('ip-addr-of-scheduler:8786')


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
Exception ignored in atexit callback: <function close_clusters at 0x124a8a320>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/spec.py", line 677, in close_clusters
    main_content = runpy.run_path(main_path,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/mertbozkir/Desktop/Code/lol/main.py", line 8, in <module>
    df = pd.read_json("matches.json")
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/_compat/pandas_api/latest/io.py", line 325, in read_json
    Engine.subscribe(_update_engine)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/config/pubsub.py", line 217, in subscribe
    callback(cls)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/sit    cluster.close(timeout=10)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 217, in close
e-packages/modin/pandas/__init__.py", line 161, in _update_engine
    return self.sync(self._close, callback_timeout=timeout)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 339, in sync
    initialize_dask()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/modin/core/execution/dask/common/utils.py", line 47, in initialize_dask
    client = Client(n_workers=num_cpus, memory_limit=worker_memory_limit)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 988, in __init__
    return sync(
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 406, in sync
    raise exc.with_traceback(tb)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 379, in f
    self.start(timeout=timeout)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/client.py", line 1185, in start
    result = yield future
  File "/Users/mertbozkir/Desktop/Code/l    sync(self.loop, self._start, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 402, in sync
ol/venv_lol/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    wait(10)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 391, in wait
    value = future.result()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return e.wait(timeout)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/threading.py", line 607, in wait
    return fut.result()
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/spec.py", line 437, in _close
    assert w.status in {
    signaled = self._cond.wait(timeout)
AssertionError: Status.running
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/threading.py", line 324, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
    sync(self.loop, self._start, **kwargs)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 402, in sync
    wait(10)
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 391, in wait
    return e.wait(timeout)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/threading.py", line 607, in wait
Task was destroyed but it is pending!
task: <Task pending name='Task-25' coro=<_wrap_awaitable() running at /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py:650> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-23' coro=<_wrap_awaitable() running at /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py:650> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-21' coro=<_wrap_awaitable() running at /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py:650> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-26' coro=<_wrap_awaitable() running at /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py:650> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-24' coro=<_wrap_awaitable() running at /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/asyncio/tasks.py:650> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-18' coro=<Cluster._sync_cluster_info() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/deploy/cluster.py:179> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-347' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-180' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-264' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-183' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-275' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-351' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-271' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-185' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-353' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-352' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-181' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-269' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-345' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-266' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Task was destroyed but it is pending!
task: <Task pending name='Task-184' coro=<Nanny._on_worker_exit() running at /Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py:742> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[set.remove()]>
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c8890>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c9700>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258cace0>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c9540>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258cbb50>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c85f0>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258cb530>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c9070>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c8820>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c86d0>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c9460>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258cb1b0>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258ca810>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258caea0>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <coroutine object Nanny._on_worker_exit at 0x1258c9230>
Traceback (most recent call last):
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 741, in wrapper
  File "/Users/mertbozkir/Desktop/Code/lol/venv_lol/lib/python3.10/site-packages/distributed/utils.py", line 756, in __exit__
ImportError: sys.meta_path is None, Python is likely shutting down

Dec 10 '22 17:12 mertbozkir

@mertbozkir, your error seems not related to the original issue. To avoid your problem you should put the code under if __name__ == '__main__':. See more in this Section about your issue.

Dec 12 '22 08:12 YarShev

modin modin copied to clipboard

Reading large json lines dataset fails because of rows having different columns

System information

Describe the problem

Source code / logs

modin
modin copied to clipboard