Gives warning: "Found and ignore invalid json file, you may lost some process data."

[adhadseKavida]

Additionally prints this:.

Invalid json file:
    /tmp/tmp9thv7lj2/result_24189.json
    /tmp/tmp9thv7lj2/result_24336.json
    /tmp/tmp9thv7lj2/result_24337.json
    /tmp/tmp9thv7lj2/result_24187.json
    /tmp/tmp9thv7lj2/result_24168.json

What should be solution to this, trying to visualize the final generated files gives incomplete information.

[gaogaotiantian]

Did you use Windows + multiprocessing pools?

[adhadseKavida]

I'm on Linux, Fedora, yes. It's Multiprocessing + Async code in each process.

[gaogaotiantian]

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.

[junqi-lu]

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.

---

update:

hmm, it seems the issue comes from windows. I can't reproduce it on linux.

---

@gaogaotiantian I got the same warning. The OS is windows 11. I used multiprocessing.Pool(), too. And the viztracer command is just viztracer .\demo.py.

Minimal demo is like:

import multiprocessing
import random

def worker(input):
    return random.randint(1,10)

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        inputs = (i for i in range(10))
        nums = list(pool.imap(worker, inputs))
    total = sum(nums)
    print(total)

[TTianshun]

Sorry I forgot to reply to this. The most common reason that the temp json file is not valid is because the process is shut down abruptly. For example, if there's any process.kill(). There's no way for viztracer to log the files if the process is just killed. However, I don't know what the root cause for it unless you have some minimal reproducible example.

update:

hmm, it seems the issue comes from windows. I can't reproduce it on linux.

@gaogaotiantian I got the same warning. The OS is windows 11. I used multiprocessing.Pool(), too. And the viztracer command is just viztracer .\demo.py.

Minimal demo is like:

import multiprocessing
import random

def worker(input):
    return random.randint(1,10)

if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        inputs = (i for i in range(10))
        nums = list(pool.imap(worker, inputs))
    total = sum(nums)
    print(total)

Maybe you can check this. It's possible for multiprocessing to terminate the saving process on Windows. https://github.com/gaogaotiantian/viztracer/issues/364#issuecomment-1836398679

[gaogaotiantian]

@junqi-lu , yeah multiprocessing.Pool() just won't work with VizTracer on Windows. That's how CPython works and there's little we can do about that.