norm icon indicating copy to clipboard operation
norm copied to clipboard

Published 20 hours ago • verified publisher icon USNavalResearchLaboratory

NormObjectTable::Remove(NormObject*): Assertion `__null != tree.Find(theObject->GetId().GetValuePtr(), 8*sizeof(UINT16))' failed

Open honglei opened this issue 1 year ago • 3 comments

Since the delay and async-behavior of high-level event process code and low-level NORM code.

User may call NormObjectCancel while the object already been NormSenderNode::AbortObject by NORM protocol like NormSenderNode::Sync(NormObjectId objectId), which caused object is removed twice in NormSenderNode.rx_table.

void NormSenderNode::DeleteObject(NormObject* obj)
{
    if (rx_table.Remove(obj))
    {
        rx_pending_mask.Unset(obj->GetId());
        obj->Close();
        obj->Release();
    }
}  // end NormSenderNode::DeleteObject()

honglei avatar Jun 29 '23 02:06 honglei 

18:24:59.581INFO|object.py|__init__|new obj: 140253151597552 ref:6
18:24:59.582INFO|object.py|__init__|id:17814
18:24:59.582INFO|receiver.py|proc_recver_file_event|RX_OBJECT_NEW  140253151597552 20230626_1_1_002477.txt
18:24:59.582INFO|receiver.py|proc_recver_file_event|Resource paused or not config, ignore:(1, '20230626_1_1_002477.txt')
rx_table:17773,17815,
18:24:59.583INFO|object.py|cancel|cancel obj: 140253151597552 ref:5  id:17814
python3: ../src/common/normObject.cpp:4554: bool NormObjectTable::Remove(NormObject*): Assertion `__null != tree.Find(theObject->GetId().GetValuePtr(), 8*sizeof(UINT16))' failed.

When Python recv RX_OBJECT_NEW , the object may not exist in NormSenderNode.rx_table.

honglei avatar Jun 29 '23 03:06 honglei

Another probem caused by the unprocessed messages in NormInstance.notify_queue :

Log:

17:23:28.707INFO|receiver.py|proc_recver_file_event|notifyCount: 71
17:23:28.708INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_NEW msg={'linkID': 1, 'resourceID': 11, 'fileName': '20230629_1_11_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 706103), 'bytesPending': 0, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 706103), 'senderID': 1, 'senderHandle': 2259857443808}
17:23:28.709INFO|receiver.py|proc_recver_file_event|RX_OBJECT_INFO  2259862347984 11983 20230629_1_11_028146.txt
17:23:28.710INFO|receiver.py|proc_recver_file_event|RX_OBJECT_UPDATED  2259862347984 11983 20230629_1_11_028146.txt
17:23:28.712INFO|receiver.py|proc_recver_file_event|RX_OBJECT_COMPLETED  2259862347984 11983 20230629_1_11_028146.txt
17:23:28.714INFO|os_ext.py|_move_file|E:\PythonPrj\6xx\fileStorage\tmpFiles\normTempd37832 -> E:\PythonPrj\6xx\fileStorage\recvFiles\LeiDa\ChangGui\20230629_1_11_028146.txt
17:23:28.717INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_COMPLETED msg={'linkID': 1, 'resourceID': 11, 'fileName': '20230629_1_11_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 706103), 'bytesPending': None, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 712122), 'senderID': 1, 'senderHandle': 2259857443808}
17:23:28.728INFO|receiver.py|proc_recver_file_event|file completed:(1, 11, '20230629_1_11_028146.txt') 
17:23:28.729INFO|receiver.py|proc_recver_file_event|RX_OBJECT_NEW  2259862347264 11984 20230629_1_12_028146.txt
rx_table(4):(0000020E2A4ACD20,11979),(0000020E2A4A78C0,11988),(0000020E2A4ADB30,11989),(0000020E29EE3890,12006),
17:23:28.732INFO|receiver.py|proc_recver_file_event|notifyCount: 87
17:23:28.734INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_NEW msg={'linkID': 1, 'resourceID': 12, 'fileName': '20230629_1_12_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 729101), 'bytesPending': 0, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 729101), 'senderID': 1, 'senderHandle': 2259857443808}
17:23:28.735INFO|receiver.py|proc_recver_file_event|RX_OBJECT_INFO  2259862347264 11984 20230629_1_12_028146.txt
17:23:28.737INFO|receiver.py|proc_recver_file_event|RX_OBJECT_UPDATED  2259862347264 11984 20230629_1_12_028146.txt
17:23:28.739INFO|receiver.py|proc_recver_file_event|RX_OBJECT_COMPLETED  2259862347264 11984 20230629_1_12_028146.txt
17:23:28.741INFO|os_ext.py|_move_file|E:\PythonPrj\6xx\fileStorage\tmpFiles\normTempe37832 -> E:\PythonPrj\6xx\fileStorage\recvFiles\LeiDa\DanZhang\R\20230629_1_12_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.744INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_COMPLETED msg={'linkID': 1, 'resourceID': 12, 'fileName': '20230629_1_12_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 729101), 'bytesPending': None, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 738102), 'senderID': 1, 'senderHandle': 2259857443808}
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.756INFO|receiver.py|proc_recver_file_event|file completed:(1, 12, '20230629_1_12_028146.txt') 
17:23:28.757INFO|receiver.py|proc_recver_file_event|RX_OBJECT_NEW  2259862316304 11985 20230629_1_13_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
rx_table(3):(0000020E2A4ACD20,11979),(0000020E2A4A78C0,11988),(0000020E2A4ADB30,11989),
17:23:28.760INFO|receiver.py|proc_recver_file_event|notifyCount: 101
17:23:28.761INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_NEW msg={'linkID': 1, 'resourceID': 13, 'fileName': '20230629_1_13_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 757103), 'bytesPending': 0, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 757103), 'senderID': 1, 'senderHandle': 2259857443808}
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.766INFO|receiver.py|proc_recver_file_event|RX_OBJECT_INFO  2259862316304 11985 20230629_1_13_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.768INFO|receiver.py|proc_recver_file_event|RX_OBJECT_UPDATED  2259862316304 11985 20230629_1_13_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.774INFO|receiver.py|proc_recver_file_event|RX_OBJECT_COMPLETED  2259862316304 11985 20230629_1_13_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

P17:23:28.775INFO|os_ext.py|_move_file|E:\PythonPrj\6xx\fileStorage\tmpFiles\normTempf37832 -> E:\PythonPrj\6xx\fileStorage\recvFiles\LeiDa\DangZhan\V\20230629_1_13_028146.txt
roto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.777INFO|mqtt_pub.py|publish_receiver_event|RX_OBJECT_COMPLETED msg={'linkID': 1, 'resourceID': 13, 'fileName': '20230629_1_13_028146.txt', 'fileSize': 30008, 'compressed': False, 'fileHandle': None, 'filePath': None, 'beginTime': datetime.datetime(2023, 6, 29, 17, 23, 28, 757103), 'bytesPending': None, 'now': datetime.datetime(2023, 6, 29, 17, 23, 28, 774101), 'senderID': 1, 'senderHandle': 2259857443808}
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.804INFO|receiver.py|proc_recver_file_event|file completed:(1, 13, '20230629_1_13_028146.txt') 
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
17:23:28.807INFO|receiver.py|proc_recver_file_event|RX_OBJECT_NEW  2259862345824 11986 20230629_1_14_028146.txt
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) Warning: mkstemp() error: `` 操作成功完成。

17:23:28.808INFO|receiver.py|proc_recver_file_event|Resource paused or not config, ignore:(14, '20230629_1_14_028146.txt')
Proto Fatal: Error opening file "": 系统找不到指定的路径。

Proto Fatal: NormFileObject::Open() recv file.Open() error!
Proto Error: NormInstance::Notify(RX_OBJECT_NEW) file object accept error!
Proto Error: NormSenderNode::HandleObjectMessage() object not accepted
rx_table(3):(0000020E2A4ACD20,11979),(0000020E2A4A78C0,11988),(0000020E2A4ADB30,11989),
17:23:28.809INFO|object.py|cancel|cancel obj: 2259862345824 ref:5  id:11986
Proto Error: NormObject::Release() releasing non-retained object?!

notifyCount: 87 mean the length of NormInstance.notify_queue

honglei avatar Jun 29 '23 09:06 honglei

So it may be the case I need to have a mutex associated with access to the rx_table so user application code cannot conflict with the protocol engine code? I'm not 100% certain I understand the cause of the issue but probably a concurrency problem of some sort ...I was trying to keep some of the API calls "lightweight" but may have overlooked something here. I need to investigate this further.

bebopagogo avatar Jan 24 '24 18:01 bebopagogo