CapsNet_Mxnet icon indicating copy to clipboard operation
CapsNet_Mxnet copied to clipboard

producing nan's starting from epoch 7?

Open machuck opened this issue 7 years ago • 0 comments

I tried a couple of times. Epoch number may vary but it quickly runs into [nan nan nan ...]

can@r710:~/CapNet/CapsNet_Mxnet$ python3.6 CapsNet.py  --train "True"
Namespace(batch_size=16, epochs=1, train=True)
/usr/local/lib/python3.6/site-packages/mxnet-1.0.0-py3.6.egg/mxnet/gluon/data/vision.py:118: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
  label = np.fromstring(fin.read(), dtype=np.uint8).astype(np.int32)
/usr/local/lib/python3.6/site-packages/mxnet-1.0.0-py3.6.egg/mxnet/gluon/data/vision.py:122: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
  data = np.fromstring(fin.read(), dtype=np.uint8)
====================================net====================================
Sequential(
  (0): Conv2D(None -> 256, kernel_size=(9, 9), stride=(1, 1))
  (1): PrimaryConv(
    (dim_0): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_1): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_2): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_3): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_4): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_5): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_6): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_7): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_8): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_9): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_10): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_11): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_12): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_13): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_14): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_15): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_16): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_17): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_18): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_19): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_20): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_21): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_22): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_23): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_24): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_25): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_26): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_27): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_28): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_29): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_30): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
    (dim_31): Conv2D(None -> 8, kernel_size=(9, 9), stride=(2, 2))
  )
  (2): DigitCaps(

  )
  (3): Length(

  )
)
====================================train====================================
  0%|                                         | 0/3750 [00:00<?, ?b/s]
[ 0.00053215 -0.00086554 -0.00070903 -0.00042224 -0.00170875 -0.00171388
  0.00125246  0.00120847  0.00166339  0.0001391   0.00141779  0.0007026
 -0.00180562  0.00125801  0.00105987 -0.00183327]
<NDArray 16 @cpu(0)>
  0%|                               | 1/3750 [00:05<5:14:01,  5.03s/b]
[ 0.00437646  0.00061251 -0.0017054  -0.00913203 -0.00747492 -0.00985935
  0.00951486 -0.00505854  0.00964234  0.00774467 -0.00641411  0.00596963
 -0.00305548 -0.00558549 -0.00095856  0.00287683]
<NDArray 16 @cpu(0)>
  0%|                               | 2/3750 [00:09<4:47:08,  4.60s/b]
[ 0.00695158  0.00160247 -0.00237272 -0.01496771 -0.01133777 -0.01531678
  0.01505066 -0.00925702  0.01498816  0.01284024 -0.01166136  0.00949801
 -0.00389259 -0.01017031 -0.00231048  0.00603203]
<NDArray 16 @cpu(0)>
  0%|                               | 3/3750 [00:13<4:38:14,  4.46s/b]
[ 0.00894154  0.00236739 -0.00288832 -0.01947842 -0.01432313 -0.019535
  0.01932948 -0.01250185  0.01912007  0.01677865 -0.01571707  0.01222481
 -0.0045394  -0.01371383 -0.0033551   0.00847037]
<NDArray 16 @cpu(0)>
  0%|                               | 4/3750 [00:17<4:33:27,  4.38s/b]
[ 0.01057109  0.00299369 -0.00331049 -0.02317308 -0.01676804 -0.02298999
  0.02283413 -0.01515932  0.02250434  0.02000437 -0.0190389   0.01445791
 -0.00506899 -0.01661601 -0.00421045  0.01046718]
<NDArray 16 @cpu(0)>
  0%|                               | 5/3750 [00:21<4:30:31,  4.33s/b]
[ 0.01194801  0.00352284 -0.00366715 -0.02629571 -0.01883411 -0.02590997
  0.02579609 -0.01740507  0.02536453  0.02273052 -0.02184629  0.01634494
 -0.00551642 -0.01906862 -0.00493313  0.01215449]
<NDArray 16 @cpu(0)>
  0%|                               | 6/3750 [00:25<4:28:52,  4.31s/b]
[ 0.01313437  0.00397869 -0.00397441 -0.02898684 -0.02061442 -0.02842639
  0.0283487  -0.01934025  0.0278294   0.02507982 -0.02426564  0.01797093
 -0.00590188 -0.02118212 -0.00555573  0.01360835]
<NDArray 16 @cpu(0)>
  0%|                               | 7/3750 [00:30<4:27:22,  4.29s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                               | 8/3750 [00:34<4:26:11,  4.27s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                               | 9/3750 [00:38<4:25:14,  4.25s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                              | 10/3750 [00:42<4:24:30,  4.24s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                              | 11/3750 [00:46<4:23:50,  4.23s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                              | 12/3750 [00:50<4:23:16,  4.23s/b]
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan]
<NDArray 16 @cpu(0)>
  0%|                              | 13/3750 [00:54<4:22:47,  4.22s/b]

machuck avatar Feb 12 '18 17:02 machuck