Three problems Could anyone give me some suggestion ?

[way790110]

1.I have already trained siammask_base,siammask_sharp,siamrpn_resnet.

But when i was training ,CPU memory incresing until 100% then stop

2.I tried demo.py ,but the bound box isn't rotated only Min Max Rectangle.

3.When i beginning training ,i get "[Warning] missing keys".

These situation is normal or unnormal?

Could anyone give me some suggestion?

Gmail:[email protected]

[HYOJINPARK]

I also suffer from same memory problem. Only thing what I found some methods are 1) reduce number of print log 2) reduce iteration by increase batch size

[way790110]

@HYOJINPARK Thanks your answer. Have you trained this model? Could you implement rotated bounding box? My friend and i can't implement rotated bbox.but we don't know why.

[HYOJINPARK]

@way790110 Umm?? Rotated box? That is implemented by opencv as line 276 in test.py Do you check it? In my case rotated box is no problem. The problem is memory leak. Finally the model require 100GB memory....in my case...

[way790110]

@HYOJINPARK thanks for comment ,I am trying.

[HYOJINPARK]

@way790110 And if you find any solution about memory problem. plz tell me. I also get suffer from that....

[way790110]

@HYOJINPARK I can't fix it. Did you have [warning] missing keys problem like fourth picture?

[HYOJINPARK]

@way790110 Yes I also got same missing key I checked both 0.4.1, and 1.3.0 pytorch verseion I think the resnet.model was trained under 0.4.0 Under my knowledge, from 0.4.1 pytorch they make new parameter the name is num_batches_tracked plz see this page https://stackoverflow.com/questions/53678133/load-pytorch-model-from-0-4-1-to-0-4-0

[way790110]

@HYOJINPARK I had read this page and tried 1.3.0 torch version several days before. It confuses me that we get same situation, but bounding box still can't be rotated when I used my own checkpoint to run demo.py. However,Thanks for your reply.

[HYOJINPARK]

@way790110 Hmm I try to another video, it works well for rotated box Did you check opencv ? In my case, rotated box is ok, just the problem is reproducing same accuracy from paper and memory leak

[way790110]

@HYOJINPARK I checked opencv is work. This is bounding box in my case. https://www.youtube.com/watch?v=-0KtQyZvi6Q I think this is unusual situation. :((

[jwma0725]

@HYOJINPARK Have you solved the problem of memory leak?

[SamihaSara]

@HYOJINPARK I tried to see progress on Tensorboard ( logs at SiamMask/experiments/siammasm_base/logs/) with port forwarding from server to local pc. But it did not output any graph or update. Can you please share the steps you followed to visualize in tensorboard?

[haishibei]

@xiuruisi @HYOJINPARK ,hello, I met the error"Out of memeory: Kill process score 380 or sacrifice child", have you fixed it?

[zosel260]

@HYOJINPARK , @way790110 Did you find solution or workaround for the memory leak problems? I think it is related to this issue : https://github.com/pytorch/pytorch/issues/13246 But I don't know how to resolve it. I just reduced the num of workers (multiprocessing processors), to mitigate the speed of leak. It was not fundamental solution, I'd like to know how to solve the problem. Please give me any comments. Thanks in advance.

[nanowhiter]

I think you didn't get the well-trained siammask_sharp model. May I see the final rpn_mask_loss and siammask_loss?