datawig
datawig copied to clipboard
Published
20 hours ago •
awslabs
awslabs
Question: When assigning a numeric variable
Why do we get cross-entropy and accuracy logs when we assign a numeric variable? I got a continuous value for the Impute value, but I'm wondering.
2021-02-19 19:20:06,854 [INFO] NumExpr defaulting to 8 threads.
2021-02-19 19:24:50,957 [INFO]
========== start: fit model
2021-02-19 19:24:50,957 [WARNING] Already bound, ignoring bind()
2021-02-19 19:25:18,406 [INFO] Epoch[0] Batch [0-6639] Speed: 3870.89 samples/sec cross-entropy=2.836561 total_votes-accuracy=0.000000
2021-02-19 19:25:45,889 [INFO] Epoch[0] Train-cross-entropy=2.407569
2021-02-19 19:25:45,890 [INFO] Epoch[0] Train-total_votes-accuracy=0.000000
2021-02-19 19:25:45,891 [INFO] Epoch[0] Time cost=54.932
2021-02-19 19:25:45,893 [INFO] Saved checkpoint to "imputer_model/model-0000.params"
2021-02-19 19:25:50,622 [INFO] Epoch[0] Validation-cross-entropy=2.897565
2021-02-19 19:25:50,623 [INFO] Epoch[0] Validation-total_votes-accuracy=0.000000
2021-02-19 19:26:19,453 [INFO] Epoch[1] Batch [0-6639] Speed: 3685.39 samples/sec cross-entropy=2.033162 total_votes-accuracy=0.000000
2021-02-19 19:26:46,887 [INFO] Epoch[1] Train-cross-entropy=1.915317
2021-02-19 19:26:46,888 [INFO] Epoch[1] Train-total_votes-accuracy=0.000000
2021-02-19 19:26:46,888 [INFO] Epoch[1] Time cost=56.265
2021-02-19 19:26:46,890 [INFO] Saved checkpoint to "imputer_model/model-0001.params"
2021-02-19 19:26:51,626 [INFO] Epoch[1] Validation-cross-entropy=2.187781
2021-02-19 19:26:51,626 [INFO] Epoch[1] Validation-total_votes-accuracy=0.000000
2021-02-19 19:27:19,355 [INFO] Epoch[2] Batch [0-6639] Speed: 3831.58 samples/sec cross-entropy=1.926377 total_votes-accuracy=0.000000
2021-02-19 19:27:46,809 [INFO] Epoch[2] Train-cross-entropy=1.839549
2021-02-19 19:27:46,810 [INFO] Epoch[2] Train-total_votes-accuracy=0.000000
2021-02-19 19:27:46,810 [INFO] Epoch[2] Time cost=55.183
2021-02-19 19:27:46,813 [INFO] Saved checkpoint to "imputer_model/model-0002.params"
2021-02-19 19:27:51,539 [INFO] Epoch[2] Validation-cross-entropy=2.001703
2021-02-19 19:27:51,540 [INFO] Epoch[2] Validation-total_votes-accuracy=0.000000
2021-02-19 19:28:19,027 [INFO] Epoch[3] Batch [0-6639] Speed: 3865.17 samples/sec cross-entropy=1.891239 total_votes-accuracy=0.000000
2021-02-19 19:28:46,633 [INFO] Epoch[3] Train-cross-entropy=1.813997
2021-02-19 19:28:46,634 [INFO] Epoch[3] Train-total_votes-accuracy=0.000000
2021-02-19 19:28:46,634 [INFO] Epoch[3] Time cost=55.094
2021-02-19 19:28:46,637 [INFO] Saved checkpoint to "imputer_model/model-0003.params"
2021-02-19 19:28:51,367 [INFO] Epoch[3] Validation-cross-entropy=1.956922
2021-02-19 19:28:51,368 [INFO] Epoch[3] Validation-total_votes-accuracy=0.000000
2021-02-19 19:29:18,846 [INFO] Epoch[4] Batch [0-6639] Speed: 3866.56 samples/sec cross-entropy=1.873258 total_votes-accuracy=0.000000
2021-02-19 19:29:46,276 [INFO] Epoch[4] Train-cross-entropy=1.806516
2021-02-19 19:29:46,277 [INFO] Epoch[4] Train-total_votes-accuracy=0.000000
2021-02-19 19:29:46,277 [INFO] Epoch[4] Time cost=54.909
2021-02-19 19:29:46,279 [INFO] Saved checkpoint to "imputer_model/model-0004.params"
2021-02-19 19:29:51,008 [INFO] Epoch[4] Validation-cross-entropy=1.971730
2021-02-19 19:29:51,009 [INFO] Epoch[4] Validation-total_votes-accuracy=0.000000
2021-02-19 19:30:18,524 [INFO] Epoch[5] Batch [0-6639] Speed: 3861.23 samples/sec cross-entropy=1.869224 total_votes-accuracy=0.000000
2021-02-19 19:30:46,013 [INFO] Epoch[5] Train-cross-entropy=1.799461
2021-02-19 19:30:46,014 [INFO] Epoch[5] Train-total_votes-accuracy=0.000000
2021-02-19 19:30:46,014 [INFO] Epoch[5] Time cost=55.005
2021-02-19 19:30:46,016 [INFO] Saved checkpoint to "imputer_model/model-0005.params"
2021-02-19 19:30:50,742 [INFO] Epoch[5] Validation-cross-entropy=1.914169
2021-02-19 19:30:50,743 [INFO] Epoch[5] Validation-total_votes-accuracy=0.000000
2021-02-19 19:31:18,287 [INFO] Epoch[6] Batch [0-6639] Speed: 3857.19 samples/sec cross-entropy=1.848747 total_votes-accuracy=0.000000
2021-02-19 19:31:45,981 [INFO] Epoch[6] Train-cross-entropy=1.785161
2021-02-19 19:31:45,981 [INFO] Epoch[6] Train-total_votes-accuracy=0.000000
2021-02-19 19:31:45,982 [INFO] Epoch[6] Time cost=55.238
2021-02-19 19:31:45,984 [INFO] Saved checkpoint to "imputer_model/model-0006.params"
2021-02-19 19:31:50,726 [INFO] Epoch[6] Validation-cross-entropy=1.864254
2021-02-19 19:31:50,727 [INFO] Epoch[6] Validation-total_votes-accuracy=0.000000
2021-02-19 19:32:18,360 [INFO] Epoch[7] Batch [0-6639] Speed: 3844.81 samples/sec cross-entropy=1.842331 total_votes-accuracy=0.000000
2021-02-19 19:32:45,956 [INFO] Epoch[7] Train-cross-entropy=1.781625
2021-02-19 19:32:45,957 [INFO] Epoch[7] Train-total_votes-accuracy=0.000000
2021-02-19 19:32:45,957 [INFO] Epoch[7] Time cost=55.230
2021-02-19 19:32:45,961 [INFO] Saved checkpoint to "imputer_model/model-0007.params"
2021-02-19 19:32:50,694 [INFO] Epoch[7] Validation-cross-entropy=1.862272
2021-02-19 19:32:50,695 [INFO] Epoch[7] Validation-total_votes-accuracy=0.000000
2021-02-19 19:33:18,318 [INFO] Epoch[8] Batch [0-6639] Speed: 3846.10 samples/sec cross-entropy=1.836069 total_votes-accuracy=0.000000
2021-02-19 19:33:45,916 [INFO] Epoch[8] Train-cross-entropy=1.777847
2021-02-19 19:33:45,917 [INFO] Epoch[8] Train-total_votes-accuracy=0.000000
2021-02-19 19:33:45,917 [INFO] Epoch[8] Time cost=55.222
2021-02-19 19:33:45,919 [INFO] Saved checkpoint to "imputer_model/model-0008.params"
2021-02-19 19:33:50,644 [INFO] Epoch[8] Validation-cross-entropy=1.833026
2021-02-19 19:33:50,645 [INFO] Epoch[8] Validation-total_votes-accuracy=0.000000
2021-02-19 19:34:18,208 [INFO] Epoch[9] Batch [0-6639] Speed: 3854.56 samples/sec cross-entropy=1.833520 total_votes-accuracy=0.000000
2021-02-19 19:34:45,896 [INFO] Epoch[9] Train-cross-entropy=1.776226
2021-02-19 19:34:45,897 [INFO] Epoch[9] Train-total_votes-accuracy=0.000000
2021-02-19 19:34:45,897 [INFO] Epoch[9] Time cost=55.252
2021-02-19 19:34:45,900 [INFO] Saved checkpoint to "imputer_model/model-0009.params"
2021-02-19 19:34:50,627 [INFO] Epoch[9] Validation-cross-entropy=1.813570
2021-02-19 19:34:50,628 [INFO] Epoch[9] Validation-total_votes-accuracy=0.000000
2021-02-19 19:35:18,287 [INFO] Epoch[10] Batch [0-6639] Speed: 3841.24 samples/sec cross-entropy=1.830642 total_votes-accuracy=0.000000
2021-02-19 19:35:45,914 [INFO] Epoch[10] Train-cross-entropy=1.778353
2021-02-19 19:35:45,915 [INFO] Epoch[10] Train-total_votes-accuracy=0.000000
2021-02-19 19:35:45,915 [INFO] Epoch[10] Time cost=55.287
2021-02-19 19:35:45,917 [INFO] Saved checkpoint to "imputer_model/model-0010.params"
2021-02-19 19:35:50,653 [INFO] Epoch[10] Validation-cross-entropy=1.804272
2021-02-19 19:35:50,654 [INFO] Epoch[10] Validation-total_votes-accuracy=0.000000
2021-02-19 19:36:18,289 [INFO] Epoch[11] Batch [0-6639] Speed: 3844.46 samples/sec cross-entropy=1.830434 total_votes-accuracy=0.000000
2021-02-19 19:36:45,936 [INFO] Epoch[11] Train-cross-entropy=1.775856
2021-02-19 19:36:45,937 [INFO] Epoch[11] Train-total_votes-accuracy=0.000000
2021-02-19 19:36:45,937 [INFO] Epoch[11] Time cost=55.283
2021-02-19 19:36:45,940 [INFO] Saved checkpoint to "imputer_model/model-0011.params"
2021-02-19 19:36:50,669 [INFO] Epoch[11] Validation-cross-entropy=1.835253
2021-02-19 19:36:50,670 [INFO] Epoch[11] Validation-total_votes-accuracy=0.000000
2021-02-19 19:37:18,301 [INFO] Epoch[12] Batch [0-6639] Speed: 3845.06 samples/sec cross-entropy=1.821207 total_votes-accuracy=0.000000
2021-02-19 19:37:47,084 [INFO] Epoch[12] Train-cross-entropy=1.769923
2021-02-19 19:37:47,085 [INFO] Epoch[12] Train-total_votes-accuracy=0.000000
2021-02-19 19:37:47,085 [INFO] Epoch[12] Time cost=56.415
2021-02-19 19:37:47,087 [INFO] Saved checkpoint to "imputer_model/model-0012.params"
2021-02-19 19:37:51,819 [INFO] No improvement detected for 3 epochs compared to 1.8135704350806672 last error obtained: 1.8220642169074315, stopping here
2021-02-19 19:37:51,820 [INFO]
========== done (780.864077091217 s) fit model
/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/datawig/calibration.py:92: RuntimeWarning: invalid value encountered in log
return np.log(probas)
/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/datawig/calibration.py:59: RuntimeWarning: invalid value encountered in greater_equal
bin_mask = (top_probas >= bin_lower) & (top_probas < bin_upper)
/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/datawig/calibration.py:59: RuntimeWarning: invalid value encountered in less
bin_mask = (top_probas >= bin_lower) & (top_probas < bin_upper)```
