models
models copied to clipboard
Published
20 hours ago •
tensorflow
tensorflow
CenterNet MobileNetV2 FPN 512x512 from Model Zoo cannot be trained
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [X ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- [ X] I am reporting the issue to the correct repository. (Model Garden official or research directory)
- [X ] I checked to make sure that this issue has not already been filed.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/research/...
2. Describe the bug
I am trying to fine tune the Model: CenterNet MobileNetV2 FPN 512x512. When starting the training with model_main_tf2.py, I get the following error: ....
2022-04-11 14:00:15.627724: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8302
2022-04-11 14:00:16.055656: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8302
Traceback (most recent call last):
File "model_train.py", line 121, in <module>
model_lib_v2.train_loop(
File "/usr/local/lib/python3.8/dist-packages/object_detection/model_lib_v2.py", line 605, in train_loop
load_fine_tune_checkpoint(
File "/usr/local/lib/python3.8/dist-packages/object_detection/model_lib_v2.py", line 406, in load_fine_tune_checkpoint
ckpt.restore(
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/tracking/util.py", line 846, in assert_existing_objects_matched
raise AssertionError(
AssertionError: Found 3 Python objects that were not bound to checkpointed values, likely due to changes in the Python program. Showing 3 of 3 unmatched objects: [MirroredVariable:{
0: <tf.Variable 'conv2d_2/kernel:0' shape=(3, 3, 64, 32) dtype=float32, numpy=
array([[[[-0.0241668 , -0.00558611, -0.01062491, ..., -0.0016522 ,
0.03644528, 0.00151125],
[-0.07420737, 0.02261546, -0.0662635 , ..., 0.00855529,
0.07694782, 0.07457586],
[-0.07209867, -0.04271746, -0.06098064, ..., -0.01932877,
0.04653486, -0.05471299],
...,
[-0.07226481, -0.01699883, -0.02413958, ..., -0.03361956,
-0.08052056, 0.04185221],
[ 0.06959315, -0.06525099, -0.01809257, ..., -0.04093468,
0.06531063, 0.04805657],
[-0.01273197, -0.01726726, 0.01467999, ..., 0.01712921,
-0.0441375 , 0.02121422]],
[[ 0.07771548, -0.06726607, 0.03748473, ..., 0.02546835,
0.04213063, -0.03276644],
[ 0.00024644, 0.06112898, 0.01793859, ..., -0.05360377,
0.0219656 , 0.0077854 ],
[-0.03782751, 0.0423489 , 0.0209837 , ..., -0.02516639,
-0.0219388 , 0.01189253],
...,
[ 0.0455905 , 0.0094623 , 0.01581734, ..., -0.07547592,
-0.05574934, -0.07449023],
[-0.03514185, -0.02181902, 0.02822814, ..., -0.03406123,
0.0567762 , -0.03287574],
[ 0.02773142, -0.03016061, -0.02284924, ..., 0.04147849,
0.01201093, -0.04184844]],
[[-0.07126435, -0.04382779, 0.07339344, ..., 0.05777509,
0.05723538, 0.08191297],
[ 0.0016531 , -0.08202406, 0.0610744 , ..., -0.07987114,
0.07649001, 0.02181324],
[-0.05667222, 0.07188102, -0.05195189, ..., -0.05134585,
0.07006008, 0.06553794],
...,
[-0.05296525, -0.03888711, 0.04383696, ..., -0.01741121,
0.06272963, -0.03390916],
[ 0.03267515, 0.06682659, 0.01579773, ..., 0.07719692,
0.06550469, 0.05240425],
[ 0.03250607, 0.05190618, 0.01187553, ..., 0.06124491,
-0.03691679, 0.01005546]]],
[[[-0.01482775, -0.07787522, -0.04810248, ..., 0.01951879,
-0.06276135, -0.0413961 ],
[-0.01901057, 0.00167068, 0.04207075, ..., -0.06178653,
0.04750887, 0.03498974],
[ 0.02854908, -0.06387135, 0.04354098, ..., -0.06332044,
0.05292643, 0.033041 ],
...,
[-0.03366139, 0.05476356, 0.08109554, ..., 0.06542791,
0.02946349, -0.07329816],
[-0.04799819, 0.07999028, -0.04956587, ..., 0.05857725,
0.05628101, -0.02814484],
[-0.05198711, -0.03314112, 0.07211912, ..., -0.03439214,
-0.03319409, 0.02259532]],
[[-0.00932351, 0.06385293, -0.08299541, ..., 0.01030761,
-0.03524965, 0.08138492],
[ 0.01567229, -0.03718162, -0.04982382, ..., -0.01763231,
-0.02534608, -0.01796374],
[ 0.01089177, 0.08221344, 0.07701565, ..., -0.03604529,
-0.06360775, -0.04477559],
...,
[-0.07255437, 0.0664478 , 0.03457371, ..., -0.00040915,
-0.01511524, 0.03978828],
[-0.07751143, 0.07076731, -0.04184037, ..., -0.05305298,
-0.00551351, 0.04987057],
[-0.03803537, -0.05964959, 0.06484786, ..., 0.05646446,
0.03958979, -0.0760335 ]],
[[ 0.04583362, 0.07773495, -0.01630578, ..., -0.07768997,
-0.06219258, 0.04715998],
[-0.0348079 , 0.00796743, -0.07629599, ..., 0.0379109 ,
0.00888139, 0.00232846],
[-0.07055663, 0.08198489, 0.03179183, ..., 0.04179043,
-0.06393743, 0.01659888],
...,
[-0.04799646, -0.0272515 , 0.03786993, ..., 0.01002997,
0.05125073, 0.01347866],
[ 0.0157245 , 0.04616273, 0.02051246, ..., -0.04282379,
0.04505841, 0.0187582 ],
[ 0.06829866, 0.02478588, 0.0610946 , ..., -0.03242407,
0.03097347, 0.01833109]]],
[[[ 0.06690849, 0.08305983, -0.04527013, ..., -0.01574137,
0.00142118, -0.02331819],
[-0.00684577, -0.01478446, -0.02718081, ..., -0.07040515,
0.04043797, -0.05729395],
[ 0.0040954 , 0.07826193, 0.0813944 , ..., 0.02999107,
0.02985471, -0.08278253],
...,
[ 0.07969264, 0.04968876, 0.06287415, ..., 0.03790853,
-0.06432317, -0.00771797],
[ 0.02680693, 0.04236367, 0.04727457, ..., -0.02329554,
0.02729181, -0.04398891],
[-0.00179992, 0.05517384, -0.0769226 , ..., -0.04853859,
-0.01587605, 0.03517912]],
[[-0.07829215, -0.07785239, -0.04992118, ..., -0.03867453,
0.0509424 , 0.01287606],
[ 0.04288789, -0.00289899, -0.05555133, ..., 0.00689767,
0.01124837, -0.02151344],
[ 0.03745725, 0.08227345, 0.02080842, ..., -0.01250807,
-0.06140979, -0.01787875],
...,
[ 0.02347573, -0.04740664, -0.05676868, ..., 0.03480313,
0.07877231, 0.04637457],
[-0.08140536, -0.04968564, 0.06671641, ..., 0.00867686,
-0.05726276, -0.06373256],
[ 0.07197911, 0.05530814, -0.00801287, ..., -0.05270134,
0.00449631, -0.02682173]],
[[ 0.03618312, -0.0780355 , -0.02119585, ..., -0.04909813,
-0.00188532, -0.0290645 ],
[ 0.00269846, 0.04062986, 0.00865617, ..., 0.0760409 ,
0.03463099, -0.00463287],
[ 0.0252795 , -0.07108915, 0.04959121, ..., 0.06508196,
0.04996998, -0.04771767],
...,
[-0.03258008, 0.00543547, 0.07057161, ..., -0.01780834,
0.02479184, -0.04669372],
[ 0.00526557, 0.06085805, 0.05525746, ..., -0.03395218,
0.04603151, -0.07220101],
[ 0.05897268, 0.00232182, -0.01879668, ..., -0.07788295,
-0.00704499, 0.06542627]]]], dtype=float32)>,
1: <tf.Variable 'conv2d_2/kernel/replica_1:0' shape=(3, 3, 64, 32) dtype=float32, numpy=
array([[[[-0.0241668 , -0.00558611, -0.01062491, ..., -0.0016522 ,
0.03644528, 0.00151125],
[-0.07420737, 0.02261546, -0.0662635 , ..., 0.00855529,
0.07694782, 0.07457586],
[-0.07209867, -0.04271746, -0.06098064, ..., -0.01932877,
0.04653486, -0.05471299],
...,
[-0.07226481, -0.01699883, -0.02413958, ..., -0.03361956,
-0.08052056, 0.04185221],
[ 0.06959315, -0.06525099, -0.01809257, ..., -0.04093468,
0.06531063, 0.04805657],
[-0.01273197, -0.01726726, 0.01467999, ..., 0.01712921,
-0.0441375 , 0.02121422]],
[[ 0.07771548, -0.06726607, 0.03748473, ..., 0.02546835,
0.04213063, -0.03276644],
[ 0.00024644, 0.06112898, 0.01793859, ..., -0.05360377,
0.0219656 , 0.0077854 ],
[-0.03782751, 0.0423489 , 0.0209837 , ..., -0.02516639,
-0.0219388 , 0.01189253],
...,
[ 0.0455905 , 0.0094623 , 0.01581734, ..., -0.07547592,
-0.05574934, -0.07449023],
[-0.03514185, -0.02181902, 0.02822814, ..., -0.03406123,
0.0567762 , -0.03287574],
[ 0.02773142, -0.03016061, -0.02284924, ..., 0.04147849,
0.01201093, -0.04184844]],
[[-0.07126435, -0.04382779, 0.07339344, ..., 0.05777509,
0.05723538, 0.08191297],
[ 0.0016531 , -0.08202406, 0.0610744 , ..., -0.07987114,
0.07649001, 0.02181324],
[-0.05667222, 0.07188102, -0.05195189, ..., -0.05134585,
0.07006008, 0.06553794],
...,
[-0.05296525, -0.03888711, 0.04383696, ..., -0.01741121,
0.06272963, -0.03390916],
[ 0.03267515, 0.06682659, 0.01579773, ..., 0.07719692,
0.06550469, 0.05240425],
[ 0.03250607, 0.05190618, 0.01187553, ..., 0.06124491,
-0.03691679, 0.01005546]]],
[[[-0.01482775, -0.07787522, -0.04810248, ..., 0.01951879,
-0.06276135, -0.0413961 ],
[-0.01901057, 0.00167068, 0.04207075, ..., -0.06178653,
0.04750887, 0.03498974],
[ 0.02854908, -0.06387135, 0.04354098, ..., -0.06332044,
0.05292643, 0.033041 ],
...,
[-0.03366139, 0.05476356, 0.08109554, ..., 0.06542791,
0.02946349, -0.07329816],
[-0.04799819, 0.07999028, -0.04956587, ..., 0.05857725,
0.05628101, -0.02814484],
[-0.05198711, -0.03314112, 0.07211912, ..., -0.03439214,
-0.03319409, 0.02259532]],
[[-0.00932351, 0.06385293, -0.08299541, ..., 0.01030761,
-0.03524965, 0.08138492],
[ 0.01567229, -0.03718162, -0.04982382, ..., -0.01763231,
-0.02534608, -0.01796374],
[ 0.01089177, 0.08221344, 0.07701565, ..., -0.03604529,
-0.06360775, -0.04477559],
...,
[-0.07255437, 0.0664478 , 0.03457371, ..., -0.00040915,
-0.01511524, 0.03978828],
[-0.07751143, 0.07076731, -0.04184037, ..., -0.05305298,
-0.00551351, 0.04987057],
[-0.03803537, -0.05964959, 0.06484786, ..., 0.05646446,
0.03958979, -0.0760335 ]],
[[ 0.04583362, 0.07773495, -0.01630578, ..., -0.07768997,
-0.06219258, 0.04715998],
[-0.0348079 , 0.00796743, -0.07629599, ..., 0.0379109 ,
0.00888139, 0.00232846],
[-0.07055663, 0.08198489, 0.03179183, ..., 0.04179043,
-0.06393743, 0.01659888],
...,
[-0.04799646, -0.0272515 , 0.03786993, ..., 0.01002997,
0.05125073, 0.01347866],
[ 0.0157245 , 0.04616273, 0.02051246, ..., -0.04282379,
0.04505841, 0.0187582 ],
[ 0.06829866, 0.02478588, 0.0610946 , ..., -0.03242407,
0.03097347, 0.01833109]]],
[[[ 0.06690849, 0.08305983, -0.04527013, ..., -0.01574137,
0.00142118, -0.02331819],
[-0.00684577, -0.01478446, -0.02718081, ..., -0.07040515,
0.04043797, -0.05729395],
[ 0.0040954 , 0.07826193, 0.0813944 , ..., 0.02999107,
0.02985471, -0.08278253],
...,
[ 0.07969264, 0.04968876, 0.06287415, ..., 0.03790853,
-0.06432317, -0.00771797],
[ 0.02680693, 0.04236367, 0.04727457, ..., -0.02329554,
0.02729181, -0.04398891],
[-0.00179992, 0.05517384, -0.0769226 , ..., -0.04853859,
-0.01587605, 0.03517912]],
[[-0.07829215, -0.07785239, -0.04992118, ..., -0.03867453,
0.0509424 , 0.01287606],
[ 0.04288789, -0.00289899, -0.05555133, ..., 0.00689767,
0.01124837, -0.02151344],
[ 0.03745725, 0.08227345, 0.02080842, ..., -0.01250807,
-0.06140979, -0.01787875],
...,
[ 0.02347573, -0.04740664, -0.05676868, ..., 0.03480313,
0.07877231, 0.04637457],
[-0.08140536, -0.04968564, 0.06671641, ..., 0.00867686,
-0.05726276, -0.06373256],
[ 0.07197911, 0.05530814, -0.00801287, ..., -0.05270134,
0.00449631, -0.02682173]],
[[ 0.03618312, -0.0780355 , -0.02119585, ..., -0.04909813,
-0.00188532, -0.0290645 ],
[ 0.00269846, 0.04062986, 0.00865617, ..., 0.0760409 ,
0.03463099, -0.00463287],
[ 0.0252795 , -0.07108915, 0.04959121, ..., 0.06508196,
0.04996998, -0.04771767],
...,
[-0.03258008, 0.00543547, 0.07057161, ..., -0.01780834,
0.02479184, -0.04669372],
[ 0.00526557, 0.06085805, 0.05525746, ..., -0.03395218,
0.04603151, -0.07220101],
[ 0.05897268, 0.00232182, -0.01879668, ..., -0.07788295,
-0.00704499, 0.06542627]]]], dtype=float32)>
}, MirroredVariable:{
0: <tf.Variable 'conv2d_4/kernel:0' shape=(3, 3, 32, 24) dtype=float32, numpy=
array([[[[-0.08819695, 0.08918268, -0.07207611, ..., -0.04182342,
0.00107777, -0.10352932],
[ 0.03847943, 0.0579557 , 0.0279737 , ..., 0.09765839,
0.09236672, 0.06976824],
[-0.09796128, 0.08550034, 0.03807312, ..., 0.04415695,
0.03122673, -0.09504236],
...,
[-0.08497702, -0.08599462, 0.02730215, ..., 0.06586175,
0.06957764, 0.0523654 ],
[-0.01771906, 0.09860169, -0.0170951 , ..., 0.08860607,
0.07744088, -0.05940388],
[-0.03437793, -0.03257454, -0.01187734, ..., -0.09123419,
-0.03631649, 0.0922616 ]],
[[ 0.08095508, -0.10117683, 0.10852606, ..., 0.08254404,
0.01627604, 0.00515746],
[ 0.05421928, 0.04158094, -0.10172611, ..., -0.0083661 ,
-0.05110536, -0.06966645],
[ 0.06683234, -0.06078382, -0.08701677, ..., 0.08398546,
0.06653746, -0.02279756],
...,
[-0.01618933, -0.09420503, 0.05445888, ..., 0.05731031,
0.10507338, 0.07715774],
[-0.04336084, 0.0267091 , 0.08372024, ..., 0.01962846,
-0.02556474, 0.07149187],
[ 0.07569515, 0.0219949 , -0.03776514, ..., 0.00305028,
0.05921156, 0.0035076 ]],
[[-0.04033419, -0.06093712, -0.03014396, ..., -0.0502044 ,
0.01476093, 0.08811422],
[ 0.10615734, 0.05033403, -0.007186 , ..., -0.0085445 ,
-0.05092945, -0.00899535],
[-0.10733835, 0.0924721 , 0.08665494, ..., 0.04646339,
-0.03225426, 0.04246403],
...,
[ 0.00461653, -0.04664566, -0.09217492, ..., -0.07228261,
-0.0409582 , 0.06982129],
[ 0.05016056, -0.01614555, -0.10513051, ..., -0.03746413,
0.0570034 , 0.02217967],
[ 0.10677847, -0.07322454, 0.08161496, ..., -0.0686688 ,
0.01377764, 0.07567101]]],
[[[ 0.00912274, -0.04633683, 0.10143056, ..., -0.1046942 ,
0.00116453, 0.1036521 ],
[-0.0903623 , 0.0163432 , 0.06645002, ..., -0.02223419,
0.09337538, -0.07655121],
[-0.10645613, 0.09632032, 0.07621426, ..., 0.0983967 ,
-0.10068059, -0.04806484],
...,
[ 0.09934504, -0.10200195, 0.08592334, ..., -0.06986597,
-0.03416077, 0.02367292],
[-0.01149072, -0.04846058, -0.06080728, ..., -0.08559781,
0.09224061, 0.08904838],
[-0.04625629, -0.03634506, -0.09049884, ..., 0.02326597,
0.09388191, -0.0880729 ]],
[[ 0.02300451, -0.03803706, 0.03907412, ..., 0.06621156,
-0.01765385, 0.05038349],
[-0.04778043, 0.07026737, -0.0315681 , ..., 0.02701619,
-0.04820143, -0.00412285],
[ 0.09303492, 0.09206221, 0.10236638, ..., 0.08856825,
0.05186375, 0.07432125],
...,
[-0.03967896, -0.04850402, -0.06355068, ..., 0.03205783,
0.06731134, -0.01316866],
[-0.03957095, -0.08050577, 0.04322439, ..., -0.0177206 ,
0.00544602, -0.01330794],
[ 0.03014716, -0.02779475, -0.01239771, ..., -0.08542036,
-0.07025087, -0.06127665]],
[[-0.10335154, 0.05174904, 0.08282463, ..., -0.02664175,
0.0464303 , 0.0307372 ],
[ 0.01463231, -0.01851162, -0.06246622, ..., 0.04810097,
-0.09060597, 0.01259436],
[-0.05635893, 0.05243983, 0.08391694, ..., -0.00243896,
-0.10628884, -0.07759969],
...,
[ 0.03372183, -0.02591249, 0.05529448, ..., 0.09801318,
0.05591238, -0.09896364],
[ 0.0582481 , 0.0979769 , -0.08660583, ..., 0.03799326,
-0.03269944, 0.08103769],
[ 0.06985895, 0.08025687, -0.08123275, ..., -0.01761017,
0.03927749, -0.09608586]]],
[[[-0.07146121, 0.002823 , 0.03632086, ..., -0.01331758,
0.01921495, 0.09882941],
[ 0.07643271, -0.09610657, -0.05470611, ..., -0.07578334,
-0.00176068, 0.04666369],
[-0.08575506, -0.04835457, -0.09302825, ..., -0.05361991,
0.06864645, -0.09421811],
...,
[-0.09321196, 0.03146405, 0.0852345 , ..., 0.07924122,
0.00037946, -0.10806627],
[-0.03310116, -0.0777571 , -0.02020258, ..., -0.03841665,
0.01244852, -0.08909962],
[-0.06574959, -0.06057157, -0.06864505, ..., -0.0902791 ,
0.07137837, -0.01436114]],
[[ 0.0042103 , 0.10666841, -0.07761954, ..., -0.03579567,
-0.07148238, 0.04812472],
[ 0.0135861 , -0.03836787, -0.09601212, ..., -0.09827592,
0.00496433, 0.10152458],
[ 0.0105386 , -0.07187238, 0.02262106, ..., -0.04254922,
0.01954328, -0.0460749 ],
...,
[-0.01750676, 0.07789362, 0.0142631 , ..., 0.00459478,
0.09471185, -0.03074159],
[-0.03811944, -0.01327412, -0.09853645, ..., 0.05564725,
0.07278936, -0.0675896 ],
[-0.00087403, -0.10174411, 0.04784528, ..., 0.10076038,
0.02854321, 0.05292454]],
[[-0.01483849, 0.08268446, -0.07251991, ..., 0.07281815,
0.10225148, -0.03468052],
[ 0.07931367, 0.10108689, -0.08605617, ..., -0.04226979,
0.06348466, 0.04409487],
[-0.04766376, -0.03451052, 0.00130247, ..., -0.07589427,
-0.05620384, 0.04046241],
...,
[-0.05827254, 0.06887262, 0.04607602, ..., 0.00979128,
0.08133339, -0.01468509],
[-0.05411347, -0.02822614, -0.01122755, ..., -0.03288668,
-0.07189111, 0.05041618],
[ 0.02288612, -0.02066424, 0.01524751, ..., -0.05193813,
-0.03147901, 0.07417328]]]], dtype=float32)>,
1: <tf.Variable 'conv2d_4/kernel/replica_1:0' shape=(3, 3, 32, 24) dtype=float32, numpy=
array([[[[-0.08819695, 0.08918268, -0.07207611, ..., -0.04182342,
0.00107777, -0.10352932],
[ 0.03847943, 0.0579557 , 0.0279737 , ..., 0.09765839,
0.09236672, 0.06976824],
[-0.09796128, 0.08550034, 0.03807312, ..., 0.04415695,
0.03122673, -0.09504236],
...,
[-0.08497702, -0.08599462, 0.02730215, ..., 0.06586175,
0.06957764, 0.0523654 ],
[-0.01771906, 0.09860169, -0.0170951 , ..., 0.08860607,
0.07744088, -0.05940388],
[-0.03437793, -0.03257454, -0.01187734, ..., -0.09123419,
-0.03631649, 0.0922616 ]],
[[ 0.08095508, -0.10117683, 0.10852606, ..., 0.08254404,
0.01627604, 0.00515746],
[ 0.05421928, 0.04158094, -0.10172611, ..., -0.0083661 ,
-0.05110536, -0.06966645],
[ 0.06683234, -0.06078382, -0.08701677, ..., 0.08398546,
0.06653746, -0.02279756],
...,
[-0.01618933, -0.09420503, 0.05445888, ..., 0.05731031,
0.10507338, 0.07715774],
[-0.04336084, 0.0267091 , 0.08372024, ..., 0.01962846,
-0.02556474, 0.07149187],
[ 0.07569515, 0.0219949 , -0.03776514, ..., 0.00305028,
0.05921156, 0.0035076 ]],
[[-0.04033419, -0.06093712, -0.03014396, ..., -0.0502044 ,
0.01476093, 0.08811422],
[ 0.10615734, 0.05033403, -0.007186 , ..., -0.0085445 ,
-0.05092945, -0.00899535],
[-0.10733835, 0.0924721 , 0.08665494, ..., 0.04646339,
-0.03225426, 0.04246403],
...,
[ 0.00461653, -0.04664566, -0.09217492, ..., -0.07228261,
-0.0409582 , 0.06982129],
[ 0.05016056, -0.01614555, -0.10513051, ..., -0.03746413,
0.0570034 , 0.02217967],
[ 0.10677847, -0.07322454, 0.08161496, ..., -0.0686688 ,
0.01377764, 0.07567101]]],
[[[ 0.00912274, -0.04633683, 0.10143056, ..., -0.1046942 ,
0.00116453, 0.1036521 ],
[-0.0903623 , 0.0163432 , 0.06645002, ..., -0.02223419,
0.09337538, -0.07655121],
[-0.10645613, 0.09632032, 0.07621426, ..., 0.0983967 ,
-0.10068059, -0.04806484],
...,
[ 0.09934504, -0.10200195, 0.08592334, ..., -0.06986597,
-0.03416077, 0.02367292],
[-0.01149072, -0.04846058, -0.06080728, ..., -0.08559781,
0.09224061, 0.08904838],
[-0.04625629, -0.03634506, -0.09049884, ..., 0.02326597,
0.09388191, -0.0880729 ]],
[[ 0.02300451, -0.03803706, 0.03907412, ..., 0.06621156,
-0.01765385, 0.05038349],
[-0.04778043, 0.07026737, -0.0315681 , ..., 0.02701619,
-0.04820143, -0.00412285],
[ 0.09303492, 0.09206221, 0.10236638, ..., 0.08856825,
0.05186375, 0.07432125],
...,
[-0.03967896, -0.04850402, -0.06355068, ..., 0.03205783,
0.06731134, -0.01316866],
[-0.03957095, -0.08050577, 0.04322439, ..., -0.0177206 ,
0.00544602, -0.01330794],
[ 0.03014716, -0.02779475, -0.01239771, ..., -0.08542036,
-0.07025087, -0.06127665]],
[[-0.10335154, 0.05174904, 0.08282463, ..., -0.02664175,
0.0464303 , 0.0307372 ],
[ 0.01463231, -0.01851162, -0.06246622, ..., 0.04810097,
-0.09060597, 0.01259436],
[-0.05635893, 0.05243983, 0.08391694, ..., -0.00243896,
-0.10628884, -0.07759969],
...,
[ 0.03372183, -0.02591249, 0.05529448, ..., 0.09801318,
0.05591238, -0.09896364],
[ 0.0582481 , 0.0979769 , -0.08660583, ..., 0.03799326,
-0.03269944, 0.08103769],
[ 0.06985895, 0.08025687, -0.08123275, ..., -0.01761017,
0.03927749, -0.09608586]]],
[[[-0.07146121, 0.002823 , 0.03632086, ..., -0.01331758,
0.01921495, 0.09882941],
[ 0.07643271, -0.09610657, -0.05470611, ..., -0.07578334,
-0.00176068, 0.04666369],
[-0.08575506, -0.04835457, -0.09302825, ..., -0.05361991,
0.06864645, -0.09421811],
...,
[-0.09321196, 0.03146405, 0.0852345 , ..., 0.07924122,
0.00037946, -0.10806627],
[-0.03310116, -0.0777571 , -0.02020258, ..., -0.03841665,
0.01244852, -0.08909962],
[-0.06574959, -0.06057157, -0.06864505, ..., -0.0902791 ,
0.07137837, -0.01436114]],
[[ 0.0042103 , 0.10666841, -0.07761954, ..., -0.03579567,
-0.07148238, 0.04812472],
[ 0.0135861 , -0.03836787, -0.09601212, ..., -0.09827592,
0.00496433, 0.10152458],
[ 0.0105386 , -0.07187238, 0.02262106, ..., -0.04254922,
0.01954328, -0.0460749 ],
...,
[-0.01750676, 0.07789362, 0.0142631 , ..., 0.00459478,
0.09471185, -0.03074159],
[-0.03811944, -0.01327412, -0.09853645, ..., 0.05564725,
0.07278936, -0.0675896 ],
[-0.00087403, -0.10174411, 0.04784528, ..., 0.10076038,
0.02854321, 0.05292454]],
[[-0.01483849, 0.08268446, -0.07251991, ..., 0.07281815,
0.10225148, -0.03468052],
[ 0.07931367, 0.10108689, -0.08605617, ..., -0.04226979,
0.06348466, 0.04409487],
[-0.04766376, -0.03451052, 0.00130247, ..., -0.07589427,
-0.05620384, 0.04046241],
...,
[-0.05827254, 0.06887262, 0.04607602, ..., 0.00979128,
0.08133339, -0.01468509],
[-0.05411347, -0.02822614, -0.01122755, ..., -0.03288668,
-0.07189111, 0.05041618],
[ 0.02288612, -0.02066424, 0.01524751, ..., -0.05193813,
-0.03147901, 0.07417328]]]], dtype=float32)>
}, MirroredVariable:{
0: <tf.Variable 'conv2d_6/kernel:0' shape=(3, 3, 24, 24) dtype=float32, numpy=
array([[[[ 0.08606102, 0.02133828, 0.09397852, ..., 0.05456101,
-0.07254647, -0.06369549],
[ 0.11757558, 0.07241789, 0.01236328, ..., 0.07288297,
-0.07648374, 0.03466877],
[-0.05628521, -0.07313341, -0.10498125, ..., -0.05600417,
0.05064843, 0.02011731],
...,
[-0.06966972, -0.02508086, 0.00920738, ..., 0.03887858,
0.06033074, 0.03827909],
[-0.04022147, 0.0438972 , 0.09024148, ..., 0.06107146,
0.09625001, -0.10754783],
[ 0.10913043, -0.05390351, -0.08705642, ..., -0.06221083,
-0.04446433, -0.07477621]],
[[-0.04084454, -0.08018407, 0.09556402, ..., -0.0875058 ,
0.02139983, 0.01917499],
[ 0.03131711, -0.05142244, 0.05797661, ..., 0.05583174,
0.06187054, 0.01638282],
[ 0.07483164, 0.02077938, -0.01823655, ..., -0.10484087,
-0.03045933, 0.03745686],
...,
[ 0.11191618, -0.05186482, -0.09462049, ..., -0.08288381,
-0.10063993, -0.03048886],
[ 0.09443148, -0.00616153, 0.09558975, ..., -0.11220825,
-0.07288612, -0.07887856],
[-0.00653239, 0.02400114, -0.09839741, ..., -0.11767586,
0.09312116, 0.04656693]],
[[ 0.11542679, -0.05368758, 0.06425402, ..., -0.11535053,
0.04779048, 0.02446686],
[ 0.11502043, 0.05679414, 0.08748171, ..., -0.07810418,
0.04926529, 0.04737405],
[ 0.10771757, -0.07183568, -0.10139464, ..., -0.01259072,
-0.02503694, -0.07796538],
...,
[-0.09260835, 0.04631511, 0.05655309, ..., -0.0562142 ,
-0.07553446, 0.04379327],
[ 0.00593692, 0.04826514, -0.09204787, ..., -0.00525158,
0.03753733, 0.08265921],
[ 0.06867043, -0.09078853, -0.03616073, ..., -0.06583869,
0.05472962, -0.08741669]]],
[[[-0.06346074, 0.00812883, 0.08679547, ..., 0.03330273,
0.11213417, 0.00884286],
[-0.09899264, -0.10057584, 0.10438276, ..., -0.11026298,
-0.04781507, -0.06741004],
[ 0.06081352, -0.08240718, 0.05440914, ..., -0.06913021,
-0.04428094, -0.01307738],
...,
[-0.10539755, -0.02021675, -0.08313303, ..., 0.00643745,
0.03973878, 0.05770876],
[ 0.09776738, -0.00028778, 0.10620721, ..., 0.08922721,
0.06815334, -0.00779318],
[-0.04126503, 0.04507274, -0.11464457, ..., 0.02992766,
0.03302925, -0.03896327]],
[[ 0.10276832, -0.09293251, -0.0592724 , ..., 0.06924193,
-0.00475967, -0.05489697],
[-0.06638348, 0.03197833, 0.10724024, ..., -0.09103169,
-0.01782635, -0.11245538],
[-0.08607114, 0.07461596, -0.01880781, ..., -0.07578804,
0.01247223, 0.09038001],
...,
[ 0.08374289, -0.00640635, 0.02657738, ..., 0.04391154,
-0.10066406, 0.0067678 ],
[-0.06493689, 0.10986624, 0.08702416, ..., -0.07289744,
-0.07464907, 0.07704786],
[ 0.00241451, 0.01370098, -0.11287443, ..., -0.08606898,
-0.06735064, 0.05878558]],
[[ 0.08854575, 0.09951328, -0.11536298, ..., 0.02083465,
-0.01892039, -0.04715879],
[ 0.06613869, -0.10734035, -0.03253725, ..., 0.00136966,
0.10618437, -0.01249241],
[-0.03198221, -0.11602803, 0.06850366, ..., -0.11172124,
-0.00643473, -0.0223194 ],
...,
[ 0.11771121, 0.0017462 , 0.11548116, ..., -0.01963768,
0.0574837 , -0.01550712],
[ 0.00736193, 0.11106796, 0.00910077, ..., 0.10933992,
-0.01745902, 0.00205025],
[-0.06222794, 0.00295509, -0.10939889, ..., -0.03462673,
0.01023278, -0.08311915]]],
[[[-0.11727476, -0.11319017, 0.0940126 , ..., -0.06158096,
-0.02314992, 0.01976012],
[ 0.06936174, -0.1123307 , 0.10585409, ..., 0.10405666,
0.00868959, 0.0023226 ],
[-0.02801717, 0.00064336, -0.11410009, ..., 0.08988706,
-0.07092303, 0.06164538],
...,
[-0.00089722, -0.10464391, 0.04069043, ..., -0.06205775,
0.02325413, -0.09438531],
[-0.01484592, -0.06816092, 0.10287186, ..., -0.06451688,
-0.09377562, 0.07626493],
[ 0.09365986, -0.03568666, -0.01812022, ..., -0.01397491,
0.09481003, 0.08367757]],
[[ 0.07003637, -0.09394628, 0.04224039, ..., -0.01510523,
-0.08687401, 0.0874256 ],
[ 0.10780355, -0.01739091, 0.02957346, ..., 0.06134216,
-0.02663563, -0.06702636],
[ 0.0599551 , 0.1105189 , -0.05495623, ..., 0.11070814,
-0.03733399, -0.06322025],
...,
[-0.09858634, 0.01595028, -0.00024215, ..., 0.08820804,
0.01687173, 0.05305938],
[-0.02140259, 0.08006387, -0.11315367, ..., 0.0994742 ,
-0.05786681, -0.0090081 ],
[ 0.1065352 , 0.02218742, -0.010267 , ..., -0.1126964 ,
0.10529309, -0.04942862]],
[[ 0.09339233, 0.00887241, 0.08777042, ..., -0.01416949,
0.08466785, 0.05395674],
[ 0.03663812, -0.00492587, 0.00625113, ..., -0.09485106,
0.0102768 , -0.09939419],
[ 0.10841689, -0.10143255, -0.00779295, ..., -0.07014033,
0.03011128, 0.10243434],
...,
[-0.01902745, 0.06225405, 0.0014328 , ..., 0.11042158,
0.07861657, 0.03430737],
[ 0.06838422, -0.03028055, 0.07006916, ..., -0.05145526,
-0.07132503, 0.02309871],
[ 0.09739231, -0.08286792, -0.10041023, ..., -0.08329664,
-0.06401286, -0.06800051]]]], dtype=float32)>,
1: <tf.Variable 'conv2d_6/kernel/replica_1:0' shape=(3, 3, 24, 24) dtype=float32, numpy=
array([[[[ 0.08606102, 0.02133828, 0.09397852, ..., 0.05456101,
-0.07254647, -0.06369549],
[ 0.11757558, 0.07241789, 0.01236328, ..., 0.07288297,
-0.07648374, 0.03466877],
[-0.05628521, -0.07313341, -0.10498125, ..., -0.05600417,
0.05064843, 0.02011731],
...,
[-0.06966972, -0.02508086, 0.00920738, ..., 0.03887858,
0.06033074, 0.03827909],
[-0.04022147, 0.0438972 , 0.09024148, ..., 0.06107146,
0.09625001, -0.10754783],
[ 0.10913043, -0.05390351, -0.08705642, ..., -0.06221083,
-0.04446433, -0.07477621]],
[[-0.04084454, -0.08018407, 0.09556402, ..., -0.0875058 ,
0.02139983, 0.01917499],
[ 0.03131711, -0.05142244, 0.05797661, ..., 0.05583174,
0.06187054, 0.01638282],
[ 0.07483164, 0.02077938, -0.01823655, ..., -0.10484087,
-0.03045933, 0.03745686],
...,
[ 0.11191618, -0.05186482, -0.09462049, ..., -0.08288381,
-0.10063993, -0.03048886],
[ 0.09443148, -0.00616153, 0.09558975, ..., -0.11220825,
-0.07288612, -0.07887856],
[-0.00653239, 0.02400114, -0.09839741, ..., -0.11767586,
0.09312116, 0.04656693]],
[[ 0.11542679, -0.05368758, 0.06425402, ..., -0.11535053,
0.04779048, 0.02446686],
[ 0.11502043, 0.05679414, 0.08748171, ..., -0.07810418,
0.04926529, 0.04737405],
[ 0.10771757, -0.07183568, -0.10139464, ..., -0.01259072,
-0.02503694, -0.07796538],
...,
[-0.09260835, 0.04631511, 0.05655309, ..., -0.0562142 ,
-0.07553446, 0.04379327],
[ 0.00593692, 0.04826514, -0.09204787, ..., -0.00525158,
0.03753733, 0.08265921],
[ 0.06867043, -0.09078853, -0.03616073, ..., -0.06583869,
0.05472962, -0.08741669]]],
[[[-0.06346074, 0.00812883, 0.08679547, ..., 0.03330273,
0.11213417, 0.00884286],
[-0.09899264, -0.10057584, 0.10438276, ..., -0.11026298,
-0.04781507, -0.06741004],
[ 0.06081352, -0.08240718, 0.05440914, ..., -0.06913021,
-0.04428094, -0.01307738],
...,
[-0.10539755, -0.02021675, -0.08313303, ..., 0.00643745,
0.03973878, 0.05770876],
[ 0.09776738, -0.00028778, 0.10620721, ..., 0.08922721,
0.06815334, -0.00779318],
[-0.04126503, 0.04507274, -0.11464457, ..., 0.02992766,
0.03302925, -0.03896327]],
[[ 0.10276832, -0.09293251, -0.0592724 , ..., 0.06924193,
-0.00475967, -0.05489697],
[-0.06638348, 0.03197833, 0.10724024, ..., -0.09103169,
-0.01782635, -0.11245538],
[-0.08607114, 0.07461596, -0.01880781, ..., -0.07578804,
0.01247223, 0.09038001],
...,
[ 0.08374289, -0.00640635, 0.02657738, ..., 0.04391154,
-0.10066406, 0.0067678 ],
[-0.06493689, 0.10986624, 0.08702416, ..., -0.07289744,
-0.07464907, 0.07704786],
[ 0.00241451, 0.01370098, -0.11287443, ..., -0.08606898,
-0.06735064, 0.05878558]],
[[ 0.08854575, 0.09951328, -0.11536298, ..., 0.02083465,
-0.01892039, -0.04715879],
[ 0.06613869, -0.10734035, -0.03253725, ..., 0.00136966,
0.10618437, -0.01249241],
[-0.03198221, -0.11602803, 0.06850366, ..., -0.11172124,
-0.00643473, -0.0223194 ],
...,
[ 0.11771121, 0.0017462 , 0.11548116, ..., -0.01963768,
0.0574837 , -0.01550712],
[ 0.00736193, 0.11106796, 0.00910077, ..., 0.10933992,
-0.01745902, 0.00205025],
[-0.06222794, 0.00295509, -0.10939889, ..., -0.03462673,
0.01023278, -0.08311915]]],
[[[-0.11727476, -0.11319017, 0.0940126 , ..., -0.06158096,
-0.02314992, 0.01976012],
[ 0.06936174, -0.1123307 , 0.10585409, ..., 0.10405666,
0.00868959, 0.0023226 ],
[-0.02801717, 0.00064336, -0.11410009, ..., 0.08988706,
-0.07092303, 0.06164538],
...,
[-0.00089722, -0.10464391, 0.04069043, ..., -0.06205775,
0.02325413, -0.09438531],
[-0.01484592, -0.06816092, 0.10287186, ..., -0.06451688,
-0.09377562, 0.07626493],
[ 0.09365986, -0.03568666, -0.01812022, ..., -0.01397491,
0.09481003, 0.08367757]],
[[ 0.07003637, -0.09394628, 0.04224039, ..., -0.01510523,
-0.08687401, 0.0874256 ],
[ 0.10780355, -0.01739091, 0.02957346, ..., 0.06134216,
-0.02663563, -0.06702636],
[ 0.0599551 , 0.1105189 , -0.05495623, ..., 0.11070814,
-0.03733399, -0.06322025],
...,
[-0.09858634, 0.01595028, -0.00024215, ..., 0.08820804,
0.01687173, 0.05305938],
[-0.02140259, 0.08006387, -0.11315367, ..., 0.0994742 ,
-0.05786681, -0.0090081 ],
[ 0.1065352 , 0.02218742, -0.010267 , ..., -0.1126964 ,
0.10529309, -0.04942862]],
[[ 0.09339233, 0.00887241, 0.08777042, ..., -0.01416949,
0.08466785, 0.05395674],
[ 0.03663812, -0.00492587, 0.00625113, ..., -0.09485106,
0.0102768 , -0.09939419],
[ 0.10841689, -0.10143255, -0.00779295, ..., -0.07014033,
0.03011128, 0.10243434],
...,
[-0.01902745, 0.06225405, 0.0014328 , ..., 0.11042158,
0.07861657, 0.03430737],
[ 0.06838422, -0.03028055, 0.07006916, ..., -0.05145526,
-0.07132503, 0.02309871],
[ 0.09739231, -0.08286792, -0.10041023, ..., -0.08329664,
-0.06401286, -0.06800051]]]], dtype=float32)>
}]
....
I also had to remove all entries from the pipeline.config of the pretrained model with 'keypoint' in it.
3. Steps to reproduce
Install Object Detection API and download pretrained centernet_mobilenetv2fpn_512x512_coco17_od from model zoo. Then train the model on the coco dataset and set
fine_tune_checkpoint_version: V2
fine_tune_checkpoint: "/data/result_model_test/checkpoint/ckpt-301"
fine_tune_checkpoint_type: "detection"
in the pipeline.config file
4. Expected behavior
I expected the model to train with the given pipeline.config file and the pretrained checkpoint.
5. Additional context
Include any logs that would be helpful to diagnose the problem.
6. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
- TensorFlow installed from (source or binary): source
- TensorFlow version (use command below): 2.8.0
- Python version: 3.8.10
- CUDA/cuDNN version: 11.6
- GPU model and memory: Tesla V100 / 16GB
You should change config for using spearable_conv2d instead of conv2d when build feature_extractor. New config file should like this:
model {
center_net {
num_classes: 90
feature_extractor {
type: "mobilenet_v2_fpn_sep_conv"
use_separable_conv: true
}
}
}
Check source code for more details: https://github.com/tensorflow/models/blob/master/research/object_detection/models/center_net_mobilenet_v2_fpn_feature_extractor.py#L104
You should change config for using spearable_conv2d instead of conv2d when build feature_extractor. New config file should like this:
model { center_net { num_classes: 90 feature_extractor { type: "mobilenet_v2_fpn_sep_conv" use_separable_conv: true } } }Check source code for more details: https://github.com/tensorflow/models/blob/master/research/object_detection/models/center_net_mobilenet_v2_fpn_feature_extractor.py#L104
thank you very much for you answer although I stopped using the framework since two months ago, I appreciated it
Is this issue solved now?? CenterNet MobileNetV2 FPN 512x512 is trainable now?
