keras_word2vec.py doesn't converge

[romeokienzler]

Hi, please see below - it doesn't converge...can you confirm it is still working on your side? fyi - running on python 2.7, keras 2.1.4, TensorFlow 1.5

Iteration 75600, loss=0.764099955559 Iteration 75700, loss=0.703801393509 Iteration 75800, loss=0.786265671253 Iteration 75900, loss=0.683647096157 Iteration 76000, loss=0.696249544621 Iteration 76100, loss=0.686785459518 Iteration 76200, loss=0.641143143177 Iteration 76300, loss=0.66484028101 Iteration 76400, loss=0.673577725887 Iteration 76500, loss=0.706019639969 Iteration 76600, loss=0.662336111069 Iteration 76700, loss=0.693303346634 Iteration 76800, loss=0.667164921761 Iteration 76900, loss=0.699192345142 Iteration 77000, loss=0.666543602943 Iteration 77100, loss=0.750823795795 Iteration 77200, loss=0.216024711728 Iteration 77300, loss=0.774852216244 Iteration 77400, loss=0.725894510746 Iteration 77500, loss=0.800480008125 Iteration 77600, loss=0.599587082863 Iteration 77700, loss=0.712833166122 Iteration 77800, loss=0.707559227943 Iteration 77900, loss=0.757015168667 Iteration 78000, loss=0.791208267212 Iteration 78100, loss=0.773834228516 Iteration 78200, loss=0.702738165855 Iteration 78300, loss=0.701055109501 Iteration 78400, loss=0.707786381245 Iteration 78500, loss=0.702587544918 Iteration 78600, loss=1.01350021362 Iteration 78700, loss=0.684418618679 Iteration 78800, loss=0.670415282249 Iteration 78900, loss=0.692475199699 Iteration 79000, loss=0.699831724167 Iteration 79100, loss=0.66406583786 Iteration 79200, loss=0.572940707207 Iteration 79300, loss=0.687783002853 Iteration 79400, loss=0.693389892578 Iteration 79500, loss=0.518278539181 Iteration 79600, loss=0.726815402508 Iteration 79700, loss=0.696648955345 Iteration 79800, loss=0.739838123322 Iteration 79900, loss=0.70836687088 Iteration 80000, loss=0.712565600872 Nearest to however: idle, paperback, boris, consolidated, preserved, protest, africans, pointing, Nearest to four: bremen, vi, fire, designations, citing, ruth, flash, flanders, Nearest to such: mathrm, adaptive, urban, places, radio, exhibit, corporate, meets, Nearest to world: shelter, elite, jet, protons, evident, somalia, original, democrats, Nearest to were: clan, expectancy, comprises, compiler, persians, maxwell, defining, allah, Nearest to eight: involves, d, hearts, bit, apart, player, press, orthodox, Nearest to that: prevention, warrior, include, treaty, congo, belief, aerospace, dia, Nearest to can: waterways, gwh, chord, marriages, rituals, crossing, defended, known, Nearest to while: dial, exhibits, selective, leonard, extensions, concern, perfectly, egyptian, Nearest to or: eventually, heard, organised, mirrors, piano, blessed, touch, crowd, Nearest to after: lab, track, eritrea, implemented, fl, papacy, history, mpeg, Nearest to first: cp, obvious, demons, allowing, libya, watching, prototype, mistake, Nearest to use: corner, greenwich, neighbours, biology, converted, armenia, superhero, welsh,

Iteration 190000, loss=1.28259468079 Nearest to however: africans, lack, pressed, voted, consolidated, navigator, developed, absinthe, Nearest to four: january, vi, deeply, flash, fire, creating, overcome, implementations, Nearest to such: known, to, mathrm, and, of, radio, the, in, Nearest to world: separation, background, original, conditions, dimensional, course, verb, characters, Nearest to were: a, of, to, in, and, one, the, is, Nearest to eight: hearts, referred, herself, hamilton, census, laser, cameron, arms, Nearest to that: in, of, the, and, for, a, to, as, Nearest to can: rico, chord, scheduled, planted, costume, reflects, asks, tied, Nearest to while: darker, newspapers, loans, crusade, played, method, structural, variable, Nearest to or: the, in, and, one, on, of, to, a, Nearest to after: continuity, musician, unique, bengal, hormones, center, fairy, publishers, Nearest to first: governed, obvious, original, chemicals, fairly, evaluation, cp, aa, Nearest to use: corner, guitarist, buddha, fauna, arising, electors, god, painters, Nearest to used: legion, sphere, households, karaoke, ask, nl, raf, persons, Nearest to people: canterbury, evaluation, kingdom, loyalty, renowned, province, joel, catcher, Nearest to called: mobility, costa, shorter, labels, manpower, continued, eve, mother,

[kevkid]

I am experiencing the same thing.

[yiyang-yu]

Hi, thanks for the nice tutorial! While running the tutorial keras_word2vec (on python 3.5, keras 2.2.2, TensorFlow 1.10.1), I observe similar issue: the loss does not converge, however the model seems to ''learn'' something.

Nearest to with: the, of, or, a, that, which, an, and,
Nearest to one: nine, three, two, zero, eight, five, four, in,
Nearest to were: that, to, by, it, the, which, have, they,
Nearest to b: three, nine, eight, zero, five, one, two, f,
Nearest to years: first, nine, one, his, not, he, four, for,
Nearest to UNK: dish, ta, equipment, orwell, structural, when, public, reading,
Nearest to for: the, a, as, was, of, are, by, and,
Nearest to war: for, civil, a, from, as, of, to, its,
Nearest to to: as, that, s, a, and, for, from, are,
Nearest to there: are, the, have, a, of, on, that, for,
Nearest to into: the, it, that, which, have, a, are, as,
Nearest to their: was, is, are, that, a, in, from, have,
Nearest to such: to, as, are, it, that, the, is, s,
Nearest to or: the, with, have, a, that, they, to, of,
Nearest to they: the, as, a, of, it, in, to, and,
Nearest to however: the, from, as, according, by, in, to, that,
Iteration 190100, loss=0.44590264558792114
Iteration 190200, loss=0.0012999874306842685
Iteration 190300, loss=0.2146451771259308
Iteration 190400, loss=0.41328415274620056
Iteration 190500, loss=0.7217100262641907
Iteration 190600, loss=0.04550590738654137
Iteration 190700, loss=0.3372303247451782
Iteration 190800, loss=0.2602285146713257
Iteration 190900, loss=1.4481207132339478
Iteration 191000, loss=0.46256768703460693
Iteration 191100, loss=1.2401529550552368
Iteration 191200, loss=0.40826141834259033
Iteration 191300, loss=0.392622709274292
Iteration 191400, loss=0.9811886548995972
Iteration 191500, loss=2.449575662612915
Iteration 191600, loss=0.3673916459083557
Iteration 191700, loss=10.420924186706543
Iteration 191800, loss=0.5116040706634521
Iteration 191900, loss=1.1920999895664863e-05
Iteration 192000, loss=0.35790348052978516
Iteration 192100, loss=0.4350074827671051
Iteration 192200, loss=0.24862568080425262
Iteration 192300, loss=0.5650964975357056
Iteration 192400, loss=1.192093321833454e-07
Iteration 192500, loss=0.6832888126373291
Iteration 192600, loss=0.4801647663116455
Iteration 192700, loss=0.39242610335350037
Iteration 192800, loss=0.3397887349128723
Iteration 192900, loss=0.2602880299091339
Iteration 193000, loss=0.3896676301956177
Iteration 193100, loss=1.311302526119107e-06
Iteration 193200, loss=3.5762778338721546e-07
Iteration 193300, loss=0.6029363870620728
Iteration 193400, loss=0.2812001705169678
Iteration 193500, loss=0.5718764066696167
Iteration 193600, loss=0.016509223729372025
Iteration 193700, loss=0.3993978500366211
Iteration 193800, loss=0.4014536142349243
Iteration 193900, loss=0.3338930308818817
Iteration 194000, loss=0.5531206727027893
Iteration 194100, loss=1.192093321833454e-07
Iteration 194200, loss=0.9629538655281067
Iteration 194300, loss=2.3841855067985307e-07
Iteration 194400, loss=1.2901172637939453
Iteration 194500, loss=1.210584044456482
Iteration 194600, loss=0.5921065211296082
Iteration 194700, loss=0.6023688316345215
Iteration 194800, loss=0.7058825492858887
Iteration 194900, loss=0.41253331303596497
Iteration 195000, loss=0.29548874497413635
Iteration 195100, loss=1.192093321833454e-07
Iteration 195200, loss=0.4237133860588074
Iteration 195300, loss=0.5737499594688416
Iteration 195400, loss=2.592007875442505
Iteration 195500, loss=0.01692405715584755
Iteration 195600, loss=0.005114919506013393
Iteration 195700, loss=1.9951552152633667
Iteration 195800, loss=0.3443126976490021
Iteration 195900, loss=0.3288797438144684
Iteration 196000, loss=0.16632264852523804
Iteration 196100, loss=0.3844645619392395
Iteration 196200, loss=0.312259703874588
Iteration 196300, loss=0.0003905462799593806
Iteration 196400, loss=0.4030435383319855
Iteration 196500, loss=0.4104464650154114
Iteration 196600, loss=1.3696439266204834
Iteration 196700, loss=0.3566870391368866
Iteration 196800, loss=0.013573860749602318
Iteration 196900, loss=0.8417724370956421
Iteration 197000, loss=0.2863491475582123
Iteration 197100, loss=9.239146311301738e-05
Iteration 197200, loss=0.21538805961608887
Iteration 197300, loss=0.6012731790542603
Iteration 197400, loss=0.6248898506164551
Iteration 197500, loss=1.5272910594940186
Iteration 197600, loss=1.5872728824615479
Iteration 197700, loss=0.31945130228996277
Iteration 197800, loss=1.4567430019378662
Iteration 197900, loss=0.4676191508769989
Iteration 198000, loss=0.6201912760734558
Iteration 198100, loss=0.6517632603645325
Iteration 198200, loss=0.9002149105072021
Iteration 198300, loss=1.192093321833454e-07
Iteration 198400, loss=0.3035319745540619
Iteration 198500, loss=0.5535779595375061
Iteration 198600, loss=0.43489494919776917
Iteration 198700, loss=0.6631350517272949
Iteration 198800, loss=0.2891642153263092
Iteration 198900, loss=0.28188276290893555
Iteration 199000, loss=0.48128360509872437
Iteration 199100, loss=0.5636037588119507
Iteration 199200, loss=1.3471636772155762
Iteration 199300, loss=0.4423184394836426
Iteration 199400, loss=0.42927226424217224
Iteration 199500, loss=0.3905087411403656
Iteration 199600, loss=0.4796544313430786
Iteration 199700, loss=0.5096637010574341
Iteration 199800, loss=0.495448499917984
Iteration 199900, loss=0.5455386638641357

I am trying to understand this. Could you please confirm if it is still working on your side? Thanks!