BrokenPipeError: [Errno 32] Broken pipe

[Thinker2016]

我根据JDet/projects/retinanet/README.md文件当中的链接https://cloud.tsinghua.edu.cn/f/f12bb566d4be43bfbdc7/下载文件ckpt_30.pkl，并复制到目录JDet/projects/retinanet/work_dirs/retinanet_gaofen/checkpoints当中，运行命令 python run_net.py --config-file=configs/retinanet_gaofen.py --task=train。由于已经存在checkpoint文件，测试过程正常，但最后打包的过程报错。如下所示： Sat Sep 4 23:39:41 2021 Loading model parameters from work_dirs/retinanet_gaofen/checkpoints/ckpt_30.pkl Sat Sep 4 23:39:41 2021 Start running Sat Sep 4 23:39:41 2021 Testing... 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1126/1126 [34:38<00:00, 1.85s/it] Merge results... 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36020/36020 [18:55<00:00, 31.72it/s] 已杀死 (base) wenzhao@wenzhao-MS-7C82:~/SoftwareFactory/detection/JDet/JDet/projects/retinanet$ Process ForkPoolWorker-16: Traceback (most recent call last): File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Process ForkPoolWorker-23: Traceback (most recent call last): File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/queues.py", line 368, in put self._writer.send_bytes(obj) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/home/wenzhao/anaconda3/lib/python3.8/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe ... 谢谢。

[Thinker2016]

好像是磁盘存储空间不足。

[Thinker2016]

好像是磁盘存储空间不足。

删掉了一些文件以后又试了一下，还是出现同样的错误，不是因为磁盘存储空间不足。

[Thinker2016]

我去掉了一些测试样本之后再次试了一下，并查看了JDet测试完成后打的zip包中的xml文件，发现给出的预测包含了大量probability低于0.1的预测框，如下所示：

<annotation>
    <source>
    <filename>0.tif</filename>
    <origin>GF2/GF3</origin>
    </source>
    <research>
        <version>4.0</version>
        <provider>placeholder_affiliation</provider>
        <author>placeholder_authorname</author>
        <!--参赛课题 -->
        <pluginname>placeholder_direction</pluginname>
        <pluginclass>placeholder_suject</pluginclass>
        <time>2020-07-2020-11</time>
    </research>
    <size>
        <width>1000</width>
        <height>1000</height>
        <depth>3</depth>
    </size>
    <!--存放目标检测信息-->
    <objects>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0688</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>561.6295, 1471.9349</point>
                <point>608.5075, 1492.8219</point>
                <point>602.3088, 1506.734</point>
                <point>555.4308, 1485.84</point>
                <point>561.6295, 1471.9349</point>
            </points>
        </object>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0571</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>565.6939, 1026.0375</point>
                <point>597.6301, 1023.0295</point>
                <point>599.4427, 1042.2744</point>
                <point>567.5065, 1045.282</point>
                <point>565.6939, 1026.0375</point>
            </points>
        </object>
        <object>
            <coordinate>pixel</coordinate>
            <type>rectangle</type>
            <description>None</description>
            <possibleresult>
                <name>A330</name>                
                <probability>0.0558</probability>
            </possibleresult>
            <!--检测框坐标，首尾闭合的矩形，起始点无要求-->
            <points>  
                <point>1016.1245, 1475.4329</point>
                <point>1048.4687, 1475.2568999999999</point>
                <point>1048.5765999999999, 1495.0795</point>
                <point>1016.2324, 1495.255</point>
                <point>1016.1245, 1475.4329</point>
            </points>
        </object>

... 咱们这个工具是如何设置目标检测的probability的阈值的？

[Thinker2016]

查出来了。在test的merge阶段，程序打开了20多个python进程，导致内存溢出。 这是什么原因？

[li-xl]

merge result的时候开启了多进程,所以会有多个python进程,不想多进程可以把https://github.com/Jittor/JDet/blob/master/python/jdet/data/devkits/result_merge.py的#283行改成mergebase

[li-xl]

probabilaty的阈值可以通过设置RetinaHead的score_threshold 来指定

[Thinker2016]

好的，谢谢你。