text2image icon indicating copy to clipboard operation
text2image copied to clipboard

How much time does an epoch take for training mnist data?

Open ysjakking opened this issue 8 years ago • 0 comments

I work on a GPU: Tesla k40c with (CNMeM is enabled with initial size: 75.0% of memory, cuDNN 4007). And it takes me about 10+ min for training a epoch. Is it normal? Or I made a wrong config for my environment?

Here comes my log:

Train Results 20.4327215786 1138.28323364 1158.71595093 Validation Results building validate function 0:00:52.053928 9.35944375 752.618843994 761.978292236

Recreated Train Dataset [1139, 1308, 1155, 1142, 1335, 1308, 1309, 1304]

Epoch 1 took 0:13:34.399718 Train Results 13.6282463623 686.228092651 699.856334229 Recreated Train Dataset [1161, 1261, 1172, 1157, 1288, 1352, 1311, 1298]

Epoch 2 took 0:12:48.676186 Train Results 14.1116136597 600.27987915 614.391492615 Recreated Train Dataset [1221, 1271, 1145, 1203, 1312, 1312, 1273, 1263]

Epoch 3 took 0:11:07.076383 Train Results 12.3779268677 505.388762512 517.766689453 Validation Results building validate function 0:00:07.973637 10.1579154358 474.040048218 484.197965393

Recreated Train Dataset [1137, 1284, 1149, 1137, 1365, 1308, 1336, 1284]

Epoch 4 took 0:11:08.574547 Train Results 14.8620436646 411.504102783 426.366148071 Recreated Train Dataset [1185, 1271, 1186, 1222, 1271, 1276, 1278, 1311]

Epoch 5 took 0:10:53.159525 Train Results 13.2145661377 363.50835083 376.72291626 Recreated Train Dataset [1166, 1317, 1161, 1182, 1225, 1291, 1359, 1299]

ysjakking avatar Jul 01 '16 02:07 ysjakking