fields() from DataClasses not working properly

[jebacpogotowietylkobukmozemnieleczyc]

Hey I just wanted to ask about strange error I have. fileds() from dataclasses not working properly if dataclass is declared outside running function.

Minimalistic example:

DataClass in function

from dataclasses import dataclass, fields

import submitit


def run():
    @dataclass
    class Test:
        test: str = "test"
        u: int = 1

    t = Test()
    print(f"Test: {fields(t)}")
    return 1


executor = submitit.AutoExecutor(folder="log_test")
executor.update_parameters(timeout_min=1, slurm_partition="dev")
job = executor.submit(run)
print(job.result())
print(job.stdout())

Print:

Test: (Field(name='test',type=<class 'str'>,default='test',default_factory=<dataclasses._MISSING_TYPE object at 0x7f5bceda6cd0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD), Field(name='u',type=<class 'int'>,default=1,default_factory=<dataclasses._MISSING_TYPE object at 0x7f5bceda6cd0>,init=True,repr=True,hash=None,compare=True,metadata=mappingproxy({}),_field_type=_FIELD))

DataClass outside function

from dataclasses import dataclass, fields

import submitit

@dataclass
class Test:
    test: str = "test"
    u: int = 1

def run():
    t = Test()
    print(f"Test: {fields(t)}")
    return 1


executor = submitit.AutoExecutor(folder="log_test")
executor.update_parameters(timeout_min=1, slurm_partition="dev")
job = executor.submit(run)
print(job.result())
print(job.stdout())

Print:

Test: ()

The exact same thing happens on slurm cluster with python 3.7.4 and on my local computer without slurm with python 3.9. Is this submitit or threading issue?

[gwenzek]

Hi, thanks for reporting. Can you tell me which version of submitit and cloudpickle you're using ?

pip show submitit cloudpickle

Your bug sounds related to #30. Make sure you are using at least submitit 1.1.3

[jebacpogotowietylkobukmozemnieleczyc]

I'm using the submitit 1.2.1 and cloudpickle 1.6.0. It doesn't matter which version of submitit or python I use, it always ends the same way.

[jebacpogotowietylkobukmozemnieleczyc]

But it seems the problem comes from cloudpickle https://github.com/cloudpipe/cloudpickle/issues/386.

[gwenzek]

I think it should work if you move the dataclass to another file and that you import it.

[jgbos]

Looks like we are all hitting this issue at the same time: https://github.com/facebookresearch/hydra/issues/1621

[gwenzek]

closing since this is a cloudpickle / cpython

[ZBB-XD]

I'm a bit confused, this is happening for me still. Im using python 3.9.7 that comes with Maya 2023, and it also reproduces in my vanilla 3.12.2 python install. Do we need to go knock on cloudpickles door so that they fix this?