tensorflow-DeepFM icon indicating copy to clipboard operation
tensorflow-DeepFM copied to clipboard

收敛极度慢

Open taojian1989 opened this issue 7 years ago • 1 comments

40个左右特征,其中20+个数值型,总共500万训练数据,100万测试数据。 一个epoch大概200秒,跑到126的时候,忍受不了速度,停了,可以看得出其实一直有在收敛,但是速度很慢,有给点建议的么,各种优化算法和学习率、batch size都试过。 feature_size: 533 field_size: 38 #params: 15692 [1] train.csv-result=0.2512, valid-result=0.2515 [202.4 s] [2] train.csv-result=0.2572, valid-result=0.2561 [190.0 s] [3] train.csv-result=0.2578, valid-result=0.2568 [200.4 s] [4] train.csv-result=0.2598, valid-result=0.2583 [188.3 s] [5] train.csv-result=0.2582, valid-result=0.2569 [191.0 s] [6] train.csv-result=0.2611, valid-result=0.2598 [201.3 s] [7] train.csv-result=0.2594, valid-result=0.2578 [190.5 s] [8] train.csv-result=0.2662, valid-result=0.2648 [182.1 s] [9] train.csv-result=0.2639, valid-result=0.2623 [192.6 s] [10] train.csv-result=0.2655, valid-result=0.2639 [190.7 s] [11] train.csv-result=0.2671, valid-result=0.2654 [201.4 s] [12] train.csv-result=0.2662, valid-result=0.2644 [190.8 s] [13] train.csv-result=0.2674, valid-result=0.2653 [191.0 s] [14] train.csv-result=0.2686, valid-result=0.2665 [200.5 s] [15] train.csv-result=0.2678, valid-result=0.2657 [189.6 s] [16] train.csv-result=0.2691, valid-result=0.2664 [200.2 s] [17] train.csv-result=0.2704, valid-result=0.2676 [183.4 s] [18] train.csv-result=0.2717, valid-result=0.2692 [173.5 s] [19] train.csv-result=0.2721, valid-result=0.2688 [183.9 s] [20] train.csv-result=0.2744, valid-result=0.2709 [173.3 s] [21] train.csv-result=0.2757, valid-result=0.2718 [182.0 s] [22] train.csv-result=0.2778, valid-result=0.2734 [173.0 s] [23] train.csv-result=0.2803, valid-result=0.2753 [173.0 s] [24] train.csv-result=0.2806, valid-result=0.2747 [183.5 s] [25] train.csv-result=0.2852, valid-result=0.2792 [173.4 s] [26] train.csv-result=0.2882, valid-result=0.2813 [184.0 s] [27] train.csv-result=0.2890, valid-result=0.2818 [174.0 s] [28] train.csv-result=0.2909, valid-result=0.2834 [174.2 s] [29] train.csv-result=0.2921, valid-result=0.2841 [184.5 s] [30] train.csv-result=0.2940, valid-result=0.2860 [174.8 s] [31] train.csv-result=0.2945, valid-result=0.2862 [186.5 s] [32] train.csv-result=0.2946, valid-result=0.2859 [191.1 s] [33] train.csv-result=0.2966, valid-result=0.2879 [190.2 s] [34] train.csv-result=0.2976, valid-result=0.2888 [200.9 s] [35] train.csv-result=0.2984, valid-result=0.2892 [190.7 s] [36] train.csv-result=0.2979, valid-result=0.2885 [201.8 s] [37] train.csv-result=0.2987, valid-result=0.2896 [191.1 s] [38] train.csv-result=0.2997, valid-result=0.2899 [191.4 s] [39] train.csv-result=0.3006, valid-result=0.2910 [201.6 s] [40] train.csv-result=0.3011, valid-result=0.2914 [191.4 s] [41] train.csv-result=0.3006, valid-result=0.2902 [201.5 s] [42] train.csv-result=0.3018, valid-result=0.2919 [190.5 s] [43] train.csv-result=0.3022, valid-result=0.2912 [189.5 s] [44] train.csv-result=0.3029, valid-result=0.2926 [200.5 s] [45] train.csv-result=0.3027, valid-result=0.2923 [190.9 s] [46] train.csv-result=0.3031, valid-result=0.2922 [200.6 s] [47] train.csv-result=0.3037, valid-result=0.2930 [190.0 s] [48] train.csv-result=0.3037, valid-result=0.2929 [190.1 s] [49] train.csv-result=0.3043, valid-result=0.2929 [200.6 s] [50] train.csv-result=0.3039, valid-result=0.2932 [190.7 s] [51] train.csv-result=0.3048, valid-result=0.2939 [200.9 s] [52] train.csv-result=0.3046, valid-result=0.2926 [190.3 s] [53] train.csv-result=0.3050, valid-result=0.2936 [189.6 s] [54] train.csv-result=0.3057, valid-result=0.2940 [200.4 s] [55] train.csv-result=0.3055, valid-result=0.2944 [190.5 s] [56] train.csv-result=0.3058, valid-result=0.2943 [201.2 s] [57] train.csv-result=0.3062, valid-result=0.2948 [190.5 s] [58] train.csv-result=0.3057, valid-result=0.2948 [191.1 s] [59] train.csv-result=0.3064, valid-result=0.2949 [201.3 s] [60] train.csv-result=0.3065, valid-result=0.2944 [190.6 s] [61] train.csv-result=0.3072, valid-result=0.2955 [201.2 s] [62] train.csv-result=0.3072, valid-result=0.2954 [190.0 s] [63] train.csv-result=0.3072, valid-result=0.2955 [189.9 s] [64] train.csv-result=0.3067, valid-result=0.2944 [200.6 s] [65] train.csv-result=0.3076, valid-result=0.2958 [190.4 s] [66] train.csv-result=0.3080, valid-result=0.2964 [200.1 s] [67] train.csv-result=0.3083, valid-result=0.2960 [190.9 s] [68] train.csv-result=0.3082, valid-result=0.2963 [190.8 s] [69] train.csv-result=0.3083, valid-result=0.2957 [201.4 s] [70] train.csv-result=0.3087, valid-result=0.2964 [190.5 s] [71] train.csv-result=0.3083, valid-result=0.2966 [201.2 s] [72] train.csv-result=0.3082, valid-result=0.2963 [190.7 s] [73] train.csv-result=0.3089, valid-result=0.2966 [191.1 s] [74] train.csv-result=0.3085, valid-result=0.2957 [201.2 s] [75] train.csv-result=0.3088, valid-result=0.2971 [190.2 s] [76] train.csv-result=0.3093, valid-result=0.2974 [190.8 s] [77] train.csv-result=0.3094, valid-result=0.2973 [201.0 s] [78] train.csv-result=0.3096, valid-result=0.2971 [190.8 s] [79] train.csv-result=0.3089, valid-result=0.2969 [201.3 s] [80] train.csv-result=0.3099, valid-result=0.2972 [190.7 s] [81] train.csv-result=0.3097, valid-result=0.2969 [190.8 s] [82] train.csv-result=0.3100, valid-result=0.2974 [200.7 s] [83] train.csv-result=0.3096, valid-result=0.2970 [190.8 s] [84] train.csv-result=0.3104, valid-result=0.2979 [200.7 s] [85] train.csv-result=0.3095, valid-result=0.2974 [190.5 s] [86] train.csv-result=0.3091, valid-result=0.2967 [190.0 s] [87] train.csv-result=0.3104, valid-result=0.2982 [201.1 s] [88] train.csv-result=0.3101, valid-result=0.2967 [190.3 s] [89] train.csv-result=0.3109, valid-result=0.2981 [201.1 s] [90] train.csv-result=0.3102, valid-result=0.2977 [190.4 s] [91] train.csv-result=0.3112, valid-result=0.2982 [190.7 s] [92] train.csv-result=0.3110, valid-result=0.2980 [200.6 s] [93] train.csv-result=0.3111, valid-result=0.2980 [191.0 s] [94] train.csv-result=0.3117, valid-result=0.2985 [201.1 s] [95] train.csv-result=0.3108, valid-result=0.2977 [191.0 s] [96] train.csv-result=0.3117, valid-result=0.2989 [190.2 s] [97] train.csv-result=0.3114, valid-result=0.2982 [201.1 s] [98] train.csv-result=0.3111, valid-result=0.2985 [190.8 s] [99] train.csv-result=0.3116, valid-result=0.2980 [201.1 s] [100] train.csv-result=0.3120, valid-result=0.2984 [191.0 s] [101] train.csv-result=0.3122, valid-result=0.2984 [191.1 s] [102] train.csv-result=0.3120, valid-result=0.2985 [201.9 s] [103] train.csv-result=0.3124, valid-result=0.2992 [191.0 s] [104] train.csv-result=0.3120, valid-result=0.2989 [200.3 s] [105] train.csv-result=0.3114, valid-result=0.2980 [191.7 s] [106] train.csv-result=0.3109, valid-result=0.2974 [189.1 s] [107] train.csv-result=0.3121, valid-result=0.2986 [195.3 s] [108] train.csv-result=0.3123, valid-result=0.2991 [184.6 s] [109] train.csv-result=0.3122, valid-result=0.2993 [194.1 s] [110] train.csv-result=0.3127, valid-result=0.2995 [184.1 s] [111] train.csv-result=0.3126, valid-result=0.2986 [184.3 s] [112] train.csv-result=0.3128, valid-result=0.2995 [192.9 s] [113] train.csv-result=0.3127, valid-result=0.2986 [183.8 s] [114] train.csv-result=0.3129, valid-result=0.2990 [194.0 s] [115] train.csv-result=0.3133, valid-result=0.2994 [184.0 s] [116] train.csv-result=0.3127, valid-result=0.2989 [183.0 s] [117] train.csv-result=0.3131, valid-result=0.2997 [193.4 s] [118] train.csv-result=0.3128, valid-result=0.2991 [185.9 s] [119] train.csv-result=0.3102, valid-result=0.2969 [194.4 s] [120] train.csv-result=0.3131, valid-result=0.2997 [184.5 s] [121] train.csv-result=0.3132, valid-result=0.2988 [177.8 s] [122] train.csv-result=0.3136, valid-result=0.2995 [191.2 s] [123] train.csv-result=0.3136, valid-result=0.2995 [180.3 s] [124] train.csv-result=0.3139, valid-result=0.3001 [191.6 s] [125] train.csv-result=0.3136, valid-result=0.3000 [182.3 s] [126] train.csv-result=0.3139, valid-result=0.3001 [179.6 s]

taojian1989 avatar Nov 26 '18 01:11 taojian1989

请问老哥机器什么配置,我这边特征几十维,200w数据,直接就OOM了。。。

levylll avatar Jul 10 '19 02:07 levylll