simple-faster-rcnn-pytorch icon indicating copy to clipboard operation
simple-faster-rcnn-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon chenyuntc

object __array__ method not producing an array

Open hepinghu opened this issue 6 years ago • 20 comments

train epoch.. test epoch.. train epoch . . . 1671it [11:17, 2.47it/s]'>' not supported between instances of 'float' and 'NoneType' 1675it [11:18, 2.47it/s]Traceback (most recent call last): File "train.py", line 132, in fire.Fire() File "/home/hhp/.local/lib/python3.6/site-packages/fire/core.py", line 127, in Fire component_trace = _Fire(component, args, context, name) File "/home/hhp/.local/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire component, remaining_args) File "/home/hhp/.local/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable result = fn(*varargs, **kwargs) File "train.py", line 81, in train trainer.train_step(img, bbox, label, scale) File "/home/hhp/project/faster_rcnn_cervix/trainer.py", line 168, in train_step losses = self.forward(imgs, bboxes, labels, scale) File "/home/hhp/project/faster_rcnn_cervix/trainer.py", line 99, in forward self.faster_rcnn.rpn(features, img_size, scale) File "/home/hhp/.conda/envs/chainer/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call result = self.forward(*input, **kwargs) File "/home/hhp/project/faster_rcnn_cervix/model/region_proposal_network.py", line 127, in forward scale=scale) File "/home/hhp/project/faster_rcnn_cervix/model/utils/creator_tool.py", line 429, in call roi = roi[keep] ValueError: object array method not producing an array This problem occurred at the Kth epoch of the training. Thanks for you help.

hepinghu avatar Sep 25 '18 02:09 hepinghu

Met the same problem in creator_tool.py and non_maximum_suppression.py. Would be very grateful if there is any help or useful information

SystemErrorWang avatar Sep 26 '18 14:09 SystemErrorWang

Is this problem solved? @hepinghu @SystemErrorWang

ChuckGithub avatar Oct 10 '18 10:10 ChuckGithub

I also met this problem. And when I check the code, I found that the loc produced from rpn is -inf. I really don't konw where the errors exist

foreverzzx avatar Oct 29 '18 02:10 foreverzzx

not yet, sorry

来自 魅族 PRO 6s

-------- 原始邮件 -------- 发件人：ChuckGithub [email protected] 时间：周三 10月10日 18:38 收件人：chenyuntc/simple-faster-rcnn-pytorch [email protected] 抄送：hepinghu [email protected],Author [email protected] 主题：Re: [chenyuntc/simple-faster-rcnn-pytorch] object array method not producing an array (#72)

Is this problem solved?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/chenyuntc/simple-faster-rcnn-pytorch","title":"chenyuntc/simple-faster-rcnn-pytorch","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/chenyuntc/simple-faster-rcnn-pytorch"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ChuckGithub in #72: Is this problem solved?"}],"action":{"name":"View Issue","url":"https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-428524450"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-428524450", "url": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-428524450", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } }, { "@type": "MessageCard", "@context": "http://schema.org/extensions", "hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [chenyuntc/simple-faster-rcnn-pytorch] object array method not producing an array (#72)", "sections": [ { "text": "", "activityTitle": "ChuckGithub", "activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png", "activitySubtitle": "@ChuckGithub", "facts": [ ] } ], "potentialAction": [ { "name": "Add a comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true, "@type": "TextInput", "id": "IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "IssueComment",\n"repositoryFullName": "chenyuntc/simple-faster-rcnn-pytorch",\n"issueId": 72,\n"IssueComment": "{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "IssueClose",\n"repositoryFullName": "chenyuntc/simple-faster-rcnn-pytorch",\n"issueId": 72\n}" }, { "targets": [ { "os": "default", "uri": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-428524450" } ], "@type": "OpenUri", "name": "View on GitHub" }, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "MuteNotification",\n"threadId": 383717472\n}" } ], "themeColor": "26292E" } ]

hepinghu avatar Oct 29 '18 05:10 hepinghu

I changed the pytorch version 0.4 it’s ok

ChuckGithub avatar Oct 29 '18 05:10 ChuckGithub

OK,thanks!

来自 魅族 PRO 6s

-------- 原始邮件 -------- 发件人：ChuckGithub [email protected] 时间：周一 10月29日 13:47 收件人：chenyuntc/simple-faster-rcnn-pytorch [email protected] 抄送：hepinghu [email protected],Mention [email protected] 主题：Re: [chenyuntc/simple-faster-rcnn-pytorch] object array method not producing an array (#72)

I changed the pytorch version 0.4 it’s ok

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/chenyuntc/simple-faster-rcnn-pytorch","title":"chenyuntc/simple-faster-rcnn-pytorch","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/chenyuntc/simple-faster-rcnn-pytorch"}},"updates":{"snippets":[{"icon":"PERSON","message":"@ChuckGithub in #72: I changed the pytorch version 0.4 it’s ok "}],"action":{"name":"View Issue","url":"https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-433795089"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-433795089", "url": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-433795089", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } }, { "@type": "MessageCard", "@context": "http://schema.org/extensions", "hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [chenyuntc/simple-faster-rcnn-pytorch] object array method not producing an array (#72)", "sections": [ { "text": "", "activityTitle": "ChuckGithub", "activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png", "activitySubtitle": "@ChuckGithub", "facts": [ ] } ], "potentialAction": [ { "name": "Add a comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true, "@type": "TextInput", "id": "IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "IssueComment",\n"repositoryFullName": "chenyuntc/simple-faster-rcnn-pytorch",\n"issueId": 72,\n"IssueComment": "{{IssueComment.value}}"\n}" } ] }, { "name": "Close issue", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "IssueClose",\n"repositoryFullName": "chenyuntc/simple-faster-rcnn-pytorch",\n"issueId": 72\n}" }, { "targets": [ { "os": "default", "uri": "https://github.com/chenyuntc/simple-faster-rcnn-pytorch/issues/72#issuecomment-433795089" } ], "@type": "OpenUri", "name": "View on GitHub" }, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n"commandName": "MuteNotification",\n"threadId": 383717472\n}" } ], "themeColor": "26292E" } ]

hepinghu avatar Oct 29 '18 06:10 hepinghu

@hepinghu @ChuckGithub @foreverzzx Any solution to this? My PyTorch version is 0.4 but I am still having this issue.

abhishekcvedia avatar Nov 05 '18 16:11 abhishekcvedia

sry

ChuckGithub avatar Nov 08 '18 09:11 ChuckGithub

Hey guys, I was having the same issue. The Proposal Creator was removing all roi's due to them being smaller than the min_size. Therefore, no roi's were being fed to the RPN which led to this error.

I later found out that my pre-processing was being done incorrectly which was leading into unreasonable values at that point and everything was getting pruned (due to my pretrained network). So make sure you are performing the image correct pre-processing.

Hope this helps!

PedroCastro avatar Nov 14 '18 15:11 PedroCastro

Hey guys, I was having the same issue. The Proposal Creator was removing all roi's due to them being smaller than the min_size. Therefore, no roi's were being fed to the RPN which led to this error.

I later found out that my pre-processing was being done incorrectly which was leading into unreasonable values at that point and everything was getting pruned (due to my pretrained network). So make sure you are performing the image correct pre-processing.

Hope this helps!

thank you! it works!

AngelaDevHao avatar Nov 25 '18 07:11 AngelaDevHao

I also met this problem and my PyTorch version is 0.4.1. @F0lha @AngelaDevHao I didn't change the source code. So how could the pre-processing been done incorrectly? Can you show me some more details? Many Thanks!

blateyang avatar Dec 09 '18 05:12 blateyang

TL;DR: Make sure the width and height of your bboxes are greater than zero!

So, I also ran into this issues (I am experimenting with my own dataset and using pytorch 1.0.0)

As @foreverzzx mentioned, this error is caused by the loc produced from rpn is filled with -inf and nan.

This is the my tracing process:

  1. I further traced the code and found that it's actually produced by x (rpn's input), which is all nan, and this is caused by the VGG16 feature extractor. (Caused by the 10th layer conv layer and later conv layers in my case).
  2. I checked the problematic conv layer's weight and it all becomes nan, and this may caused by the optimization bug.
  3. I logged the losses in trainer.py to check whether there's nan or -inf, and found that rpn_loc_loss has nans.
  4. Further, I found that gt_rpn_loc produced by gt_rpn_loc, gt_rpn_label = anchor_target_creator contains some -inf.
  5. I found that this is caused by bbox2loc() in bbox_tool.py which computes the offset and scales given the source and target bboxes.
  6. In my case, I found that some of my bbox's heights (base_height in bbox2loc()) are zeros, which is the problem.
  7. Finally I made sure that all the bboxes have no zero heights or weights, which solves the bug.

Hope my trace can help you guys. @blateyang

howardyclo avatar Dec 11 '18 10:12 howardyclo

In my case, it caused by bbox generating nan sometimes. Changing lr from 1e-3 to 1e-4 will solve the problem.

HogFeet avatar Dec 15 '18 05:12 HogFeet

Thanks for advice from @howardyclo and @HogFeet . I made sure that all the bboxes have valid heights or weights and I also changed lr from 1e-3 to 1e-4. Finally the problem have been solved.

blateyang avatar Dec 21 '18 04:12 blateyang

Thank you, @howardyclo ! I changed

height = src_bbox[:, 2] - src_bbox[:, 0]
width = src_bbox[:, 3] - src_bbox[:, 1]

base_height = dst_bbox[:, 2] - dst_bbox[:, 0]
base_width = dst_bbox[:, 3] - dst_bbox[:, 1]

to

height = src_bbox[:, 2] - src_bbox[:, 0] + 1
width = src_bbox[:, 3] - src_bbox[:, 1] + 1

base_height = dst_bbox[:, 2] - dst_bbox[:, 0] + 1
base_width = dst_bbox[:, 3] - dst_bbox[:, 1] + 1

in function bbox2loc() in bbox_tool.py and the problem is solved.

zjuPeco avatar Jan 15 '19 01:01 zjuPeco

try to replace function _smooth_l1_loss with this code.

def _smooth_l1_loss(x, t, in_weight, sigma):
    sigma2 = sigma ** 2
    diff = in_weight * (x - t)
    abs_diff = diff.abs()

    flag = (abs_diff.data < (1. / sigma2)).float()
    y = (flag * (sigma2 / 2.) * (diff ** 2) +
         (1 - flag) * (abs_diff - 0.5 / sigma2))
    modif = tonumpy(y)
    modif[np.isnan(modif)] = 0
    return modif.sum()

The problem caused by nan value of loss function. Some values of tensor y becomes nan. So I just replaced nan values with zeros. Then I just sum the rest of tensor values and get positive number of Loss function which is also positive but isn't nan. Cant's say if it's sciencely correct, but training don't crush anymore.

pazlvbanke avatar Nov 16 '19 16:11 pazlvbanke

try to replace function _smooth_l1_loss with this code.

def _smooth_l1_loss(x, t, in_weight, sigma):
    sigma2 = sigma ** 2
    diff = in_weight * (x - t)
    abs_diff = diff.abs()

    flag = (abs_diff.data < (1. / sigma2)).float()
    y = (flag * (sigma2 / 2.) * (diff ** 2) +
         (1 - flag) * (abs_diff - 0.5 / sigma2))
    modif = tonumpy(y)
    modif[np.isnan(modif)] = 0
    return modif.sum()

The problem caused by nan value of loss function. Some values of tensor y becomes nan. So I just replaced nan values with zeros. Then I just sum the rest of tensor values and get positive number of Loss function which is also positive but isn't nan. Cant's say if it's sciencely correct, but training don't crush anymore.

thank you!!!!!!

Xxxxxxxi avatar Dec 12 '19 07:12 Xxxxxxxi

尝试用此代码替换函数**_smooth_l1_loss**。

def _smooth_l1_loss(x, t, in_weight, sigma):
    sigma2 = sigma ** 2
    diff = in_weight * (x - t)
    abs_diff = diff.abs()

    flag = (abs_diff.data < (1. / sigma2)).float()
    y = (flag * (sigma2 / 2.) * (diff ** 2) +
         (1 - flag) * (abs_diff - 0.5 / sigma2))
    modif = tonumpy(y)
    modif[np.isnan(modif)] = 0
    return modif.sum()

由损失函数的nan值引起的问题。张量y的某些值变为nan。所以我只是用零替换了nan值。然后我将其余的张量值求和，得到正数的Loss函数，它也是正数，但不是nan。不能说这在科学上是正确的，但是培训不再是迷恋。

谢谢！！！！！！ modif = tonumpy(y) tonumpy what's this

evanBear avatar Dec 29 '19 05:12 evanBear

I think the problem is in the ProposalCreator().This function will filter out the inappropriate anchor.So when the rpn's out is unnoraml,this will cause the roi unnormal after the loc2bbox(anchor, loc).The code after loc2bbox() will filter the anchor which area is smaller then min_size.So when the lr or loss function is not fit ,after several iterations,lot of anchor area will become unnormal and the num of anchor after filter will become 0. Finally the error will appear on the roi = roi[keep],because the len(roi) is 0. So I think we can change the LR, adjust the loss function or use the clip() to the loc.This is just my idea,.

   roi = loc2bbox(anchor, loc)

    # Clip predicted boxes to image. 
    roi[:, slice(0, 4, 2)] = np.clip(              
        roi[:, slice(0, 4, 2)], 0, img_size[0])
    roi[:, slice(1, 4, 2)] = np.clip(              
        roi[:, slice(1, 4, 2)], 0, img_size[1])

    # Remove predicted boxes with either height or width < threshold.
    min_size = self.min_size * scale       
    hs = roi[:, 2] - roi[:, 0]
    ws = roi[:, 3] - roi[:, 1]
    keep = np.where((hs >= min_size) & (ws >= min_size))[0]
    before_size = len(roi)
    roi = roi[keep, :]
    after_size = len(roi)
    score = score[keep]
    print("before_size:",before_size," after_size:",after_size)

bufferXia avatar Feb 28 '20 09:02 bufferXia

Hi guys, I also met the error when I use Tensorflow2.2-cpu. I solve the question by upgrading numpy version from 1.15.4 to 1.19.2(the latest version). Moreover, make sure that only one numpy is installed. I find that install via conda and pip will add two numpy in the system. Maybe it's also a reason.

willy2cqu avatar Nov 22 '20 12:11 willy2cqu