Incredible oversight

[michito744]

Incredible oversights occured in the following situations:

It's definitely stronger than it used to be, but it's definitely a failure to get rid of this such move.

[lightvector]

Yep, there are many dozens of other known and recorded situations where the net makes a mistake too, the same way that top human pros sometimes simply miss a move.

The thing to realize is that:

• We have not solved 19x19 and we never will fully solve it, so there will always be places you find mistakes like this.
• Unlike a human, the net does no learning on your computer, it only does learning through training. So this is loosely analogous to a top human pro arbitrary blundering and misjudging a move, except every time you try again, or have it think more, it's like you have a time machine and keep rewinding the top human pro back to the start before they knew it was a blunder. :)
• Based on the variation you show, there are actually some structural reasons why the kind of move you mention is hard for the net, even with search - it's an unsolved research problem!

Anyways, can you provide an SGF file? If you can provide an SGF file it can be used for testing or fixed via training. Thanks! :)

[michito744]

Thank you for reply.

The following game records are Shibano vs. Hane from the league of the 76th Honinbou title.

KataGo couldn't find also the 49th move. In the actual game, when W saw the 49th move he noticed the above progress and choiced another move.

(;GM[1]SZ[19]CA[Shift_JIS] PB[SHIBANO Toramaru]BR[9p] PW[HANE Naoki]WR[9p] DT[2021-04-02] RE[B+R] KM[6.5]TM[]RU[Japanese] PC[] EV[] GN[] CP[] AP[Kiin Editor 1.11] ;B[pd];W[cp];B[qp];W[dc];B[ep];W[eq];B[fq];W[dq];B[fp];W[oq] ;B[lq];W[po];B[qo];W[pn];B[rm];W[cn];B[pl];W[de];B[nc];W[jd] ;B[qf];W[dh];B[pq];W[pr];B[qr];W[pp];B[qq];W[mo];B[jp];W[ho] ;B[do];W[co];B[in];W[hq];B[gm];W[hr];B[lo];W[ln];B[lp];W[np] ;B[fr];W[fn];B[gn];W[go];B[fo];W[en];B[hn];W[gq];B[mn](;W[fl] ;B[mm];W[nr];B[iq];W[ir];B[jr];W[ol];B[om];W[pm];B[pk];W[ok] ;B[oj];W[nm];B[nk];W[nl];B[nn];W[ml];B[on];W[nj];B[pj];W[lm] ;B[ll];W[mk];B[km];W[lk];B[kn];W[mq];B[ps];W[os];B[or];W[pc] ;B[lr];W[qd];B[od];W[qe];B[qc];W[rc];B[qb];W[pf];B[rb];W[qg] ;B[cc];W[cd];B[ec];W[db];B[ed];W[dd];B[hd]) (;W[mm];B[nn];W[nm];B[kn];W[lm];B[dn]))

[michito744]

This game was very high level and difficult for AI. For example, it cost a lot to get the correct response to the 47th move.

[lightvector]

This is great, thanks. There are a few things I'm interested in testing to help the search, and this seems like a good example position to try it on.

[michito744]

Here's a new example, can you see it?

There was a failure case where the victory or defeat was overturned.

[lightvector]

Thanks for this example too! I can add it as well.

[michito744]

Thank you for reply.

The above example is a phase that did not actually appear, but it was the first step that the pros considered. KataGo has a tendency to neglect candidates that should be explored normaly.

game record:

(;GM[1]SZ[19]CA[Shift_JIS] PB[SHIBANO Toramaru]BR[9p] PW[IYAMA Yuta]WR[9p] DT[2021-07-07] RE[W+R] KM[6.5]TM[]RU[Japanese] PC[] EV[76th Hon-in-bou Title Match] GN[7th game] CP[] AP[Kiin Editor 1.11] ;B[pd];W[dd];B[qp];W[dq];B[op];W[np];B[oq];W[qc];B[pc];W[qd] ;B[qf];W[qe];B[pe];W[rf];B[qb];W[rb];B[re];W[pb];B[rc];W[qa] ;B[rd];W[qb];B[rg];W[pf];B[qg];W[nc];B[nf];W[cn];B[nn];W[ip] ;B[lc];W[ld];B[kd];W[md];B[jc];W[og];B[ng];W[oh];B[oe];W[lb] ;B[mi];W[oj];B[pi];W[qo];B[qm];W[pm];B[ql];W[nh];B[mh];W[mj] ;B[ni];W[pl];B[oi];W[rp];B[qq];W[qn];B[qk];W[rq];B[rr];W[sr] ;B[rs];W[on];B[nm];W[oo];B[no];W[nq](;B[pr];W[nr];B[pp];W[pk] ;B[kp];W[qj];B[rj];W[rm];B[rk];W[kq];B[lq];W[kr];B[lo];W[mp] ;B[lp];W[lr];B[po];W[pn];B[mb];W[kb];B[mc];W[kc];B[nb];W[ob] ;B[nd];W[ke];B[jd];W[oc];B[le];W[me];B[lf];W[ma];B[nk];W[nj] ;B[lk];W[lj];B[kk];W[kj];B[mf];W[na];B[jj];W[jk];B[jl];W[ji] ;B[ik];W[kh];B[jg];W[ii];B[fq];W[gp];B[dp];W[cp];B[cq];W[ep] ;B[do];W[eq];B[co];W[bp];B[bo];W[bq];B[jp];W[iq];B[eo];W[fp] ;B[li];W[ki];B[de];W[cd];B[ed];W[ce];B[hj];W[hh];B[df];W[ec] ;B[fc];W[bg];B[cg];W[bh];B[dc];W[eb];B[db];W[pj];B[qi];W[ig] ;B[bc];W[cf];B[fd];W[ck];B[ci];W[ch];B[dh];W[dg];B[eg];W[ie] ;B[jf];W[gg];B[di];W[bi];B[cl];W[bk];B[dk];W[if];B[je];W[bl] ;B[gf];W[ff];B[ge];W[fg];B[gi];W[hd];B[fb];W[eh];B[cg];W[mk] ) (;B[lq];W[lr];B[mr]))

[michito744]

This trend has not changed in 60b, so here is an example of the progress in exploration.

W is still good after 100k search. KataGo misinterprets this progress as good for W.

As he explores deeper, KataGo notices something strange.

Obviously, the initial evaluation values were broken.

Clearly W is bad. If the following progress is realized, W will fail, and so W avoids this progress by more searches.

There is no trace of the evaluation of 100k search.

[michito744]

Below is an example of how KataGo was decisively ruined under very difficult circumstances.

This record is a slightly older KataGo 40b, but also the latest versions (both 40b and 60b) behave the same under short time limits. Of course, if he explore enough, KataGo can avoid ruin. However in reality, KataGo made a fatal mistake in a situation where he was not given enough resources, and lost the game.

[lightvector]

Thanks, do you have an SGF for this second case?

[lightvector]

The case that you posted on July 11 it turns out was never added for training. I can add it now though.

[michito744]

Thank you for reply.

game record: (;GM[1]FF[4] SZ[19] GN[] DT[2021-01-08] PB[B] PW[W] BR[] WR[] KM[6.5]HA[0]RU[Japanese]AP[GNU Go:3.8]RE[B+R]TM[]TC[]TT[]RL[0] ;B[pd];W[dp];B[pp];W[dd];B[cc];W[dc];B[cd];W[de];B[bf];W[qq];B[qp];W[pq];B[op];W[nq];B[rp];W[mr];B[mo];W[qk];B[qi];W[ok];B[lq];W[qc];B[pc];W[qd];B[qf];W[qe];B[pe];W[rf];B[qg];W[rg];B[oi];W[id];B[mj];W[mk];B[lk];W[nj];B[ni];W[lj];B[mi];W[ml];B[kk];W[lm];B[rr];W[lr];B[jq];W[kr];B[jo];W[kn];B[fq];W[im];B[cn];W[dn];B[dm];W[en];B[co];W[cp];B[jr];W[kq];B[kp];W[lp];B[lo];W[np];B[no];W[mp];B[ko];W[fp];B[gp];W[gq];B[ep];W[fo];B[dq];W[do];B[cq];W[bq];B[br];W[bo];B[aq];W[cm];B[hp];W[cb];B[bb];W[be];B[db];W[ce];B[eb];W[di];B[ik];W[hn];B[fj];W[fc];B[dj];W[ei];B[ej];W[fb];B[ca];W[bd];B[bc];W[cj];B[fi];W[ci];B[ck];W[dl];B[dk];W[ig];B[eh];W[fh];B[eg];W[fg];B[ff];W[gf];B[cg];W[bh];B[bg];W[dg];B[dh];W[bj];B[bm];W[bn];B[cl];W[am];B[ak];W[ch];B[df];W[fe];B[cn];W[ef];B[qb];W[rb];B[rc];W[rd];B[sb];W[sc];B[ra];W[sa];B[hl];W[jk];B[jj];W[jl];B[kj];W[rj];B[ri];W[rm];B[sb];W[ap];B[ar];W[sa];B[jn];W[jm];B[sb];W[dr];B[cr];W[hq];B[ip];W[sa];B[pm];W[fr];B[eq];W[pb];B[qm];W[go];B[bp];W[rl];B[nm];W[ql];B[sk];W[sj];B[si];W[bq];B[ob];W[bl];B[pa];W[qa];B[bk];W[cm];B[ln];W[ll];B[bm];W[er];B[hr];W[cm];B[aj];W[ai];B[bm];W[gr];B[iq];W[cm];B[ah];W[ag];B[bm];W[bp];B[cs];W[lc];B[af];W[dg];B[fm];W[pb];B[an];W[cm];B[hm];W[in];B[bm];W[hs];B[ir];W[cm];B[nl];W[nk];B[bm];W[is];B[js];W[cm];B[gn];W[gl];B[gk];W[fl];B[el];W[ho];B[bm])

The stinker in this game was that the move (before 10 moves) shown below had a high win rate.

This kind of losing is very difficult to avoid, because it is very expensive to determine that the above move is not the best one based on computational power alone.

[michito744]

New extremely severe case:

E10 is bad. There are not many chances for Black to win against ko.

Game record: (;CA[UTF-8]KM[6.5]GM[1]SZ[19]GN[]PW[W]CP[]AP[Lizzie: 0.7.4]DT[2021-10-23]EV[]PB[B]RE[]PC[]TM[]CA[UTF-8];B[pd];W[dp];B[qp];W[dd];B[op];W[fq];B[cc];W[cd];B[dc];W[ed];B[ec];W[gc];B[fc];W[fd];B[gb];W[hc];B[hb];W[ck];B[fj];W[di];B[gp];W[jq];B[lq];W[gq];B[pj];W[jo];B[ic];W[qc];B[qd];W[pc];B[oc];W[ob];B[nb];W[nc];B[od];W[mb];B[pb];W[na];B[qb];W[fl];B[hk];W[gj];B[ep];W[dq];B[do];W[gk];B[cl];W[bl];B[cp];W[cq];B[cn];W[bo];B[bm];W[dl];B[dm];W[cm];B[eq];W[er];B[fp];W[hq];B[cl];W[rc];B[rb];W[cm];B[fr];W[dr];B[cl];W[qq];B[pq];W[cm];B[em];W[cl];B[bn];W[bp];B[fm];W[gm];B[gn];W[hn];B[hm];W[gl];B[in];W[ho];B[hp];W[io];B[ip];W[iq];B[jp];W[go];B[fn];W[ko];B[kp];W[hl];B[el];W[ek];B[ge];W[mo];B[gd];W[pr];B[or];W[nq];B[nr];W[qo];B[rp];W[po];B[pp];W[rj];B[ql];W[rl];B[qj];W[qk];B[pk];W[rk];B[qh];W[pl];B[ol];W[qm];B[me];W[kc];B[cf];W[ef];B[bd];W[be];B[ce];W[bc];B[ad];W[ac];B[ee];W[fe];B[ff];W[de];B[df];W[ae];B[bd];W[ad];B[eg];W[bg];B[ee];W[bd];B[ch];W[bh];B[ci];W[bi];B[cj];W[bj];B[dj];W[gf];B[gg];W[hf];B[ie];W[hg];B[gh];W[hh];B[gi];W[ef];B[fg];W[hi];B[ei];W[rf];B[re];W[rh];B[qg];W[kj];B[bf];W[af];B[bb];W[cb];B[db];W[rg];B[qf];W[mj];B[lk];W[mk];B[lm];W[ml];B[mn];W[lp];B[dk];W[al];B[im];W[il];B[jm];W[kq];B[fk];W[mq];B[jk];W[jl];B[kl];W[ik];B[kk];W[jj];B[lj];W[li];B[mi];W[kh];B[lh];W[ki];B[no];W[fo];B[eo];W[dn];B[ej];W[co];B[en];W[mr];B[pm];W[om];B[ql];W[pn];B[nm];W[pl];B[mh];W[nl];B[ok];W[mm];B[nn];W[ln];B[nj];W[kf];B[am];W[ab];B[ba];W[ag];B[jb];W[kb];B[ri];W[si];B[qi];W[ro];B[pa];W[np];B[oo];W[jd];B[id];W[rq];B[sp];W[ps])

[Zacharias030]

this is really interesting to read! thanks

[michito744]

In a recent title match, we encountered a phase where the AI was completely useless; not only KataGo, but also FineArt made a fatal mistake.

It was completely unreliable for short searches.

If I increase the search, KataGo can find the correct move (J2) to a point, but the conclusion will not be correct at all.

When I input the procedure discovered by the professional player into KataGo, he also noticed the omission and derived a disastrous evaluation value. Incidentally, all the other relatively high values are due to the horizon effect. KataGo completely misinterprets the state of the lower side, and the horizon effect continues to occur even after many searches.

[michito744]

From the same game.

In the very serious situation shown below, there is a completely different means pointed out by a professional Go player.

It's a very complex and powerful progression, although we can't be sure it's the best. KataGo was unable to discover this procedure on his own because a high saddle blocked the way during the search.

[michito744]

Game record:

(;GM[1]SZ[19]CA[UTF-8] PB[ICHIRIKI Ryo]BR[] PW[IYAMA Yuta]WR[] DT[2022-01-22] RE[] KM[6.5]TM[]RU[Japanese] PC[] EV[] GN[] CP[] AP[Kiin Editor 1.11] ;B[pd];W[dc];B[pq];W[dq];B[ce];W[do];B[dn];W[en];B[co];W[eo] ;B[ed];W[ec];B[fd];W[gc];B[cp];W[dm];B[dp];W[ep];B[cq];W[cn] ;B[eq];W[fq];B[dr];W[qo];B[pm];W[oo];B[np];W[qd];B[qc];W[qe] ;B[pc];W[qh];B[qp];W[rp];B[rq];W[no];B[mp];W[mo];B[lp];W[lo] ;B[ko];W[op];B[oq];W[kp](;B[jo];W[iq];B[km];W[ml];B[ip];W[jq] ;B[hq];W[hr];B[gq];W[gr];B[fr];W[fp];B[go];W[fs];B[er];W[mr] ;B[nr];W[lq];B[nq];W[pp];B[qq];W[ms];B[pi];W[qi];B[pk];W[lk] ;B[ph];W[pj];B[oj];W[qj];B[nk];W[pg];B[mm];W[nm];B[ll];W[mk] ;B[nl];W[om];B[kk];W[li](;B[pn];W[ln];B[lm];W[pl];B[nj];W[ol] ;B[lj];W[ro];B[dk];W[gn];B[he];W[cd];B[bd];W[de];B[dd];W[cc] ;B[cf];W[og];B[ic];W[bc];B[hn];W[gm];B[cl];W[hm];B[in];W[fj] ;B[fk];W[gk];B[fl];W[ej];B[ek];W[gj];B[ci];W[jj];B[ih];W[hi] ;B[fm];W[fn];B[jk];W[ik];B[ij];W[ii];B[ji];W[dg];B[ch];W[gd] ;B[ff];W[ge];B[hg];W[eh];B[gf];W[df];B[ef];W[bk];B[ck];W[bj] ;B[bi];W[hj];B[ki];W[bm];B[bl];W[al];B[ak];W[am];B[cj];W[bo] ;B[bp];W[ao];B[bn];W[cm];B[me];W[aj];B[ai];W[ak]) (;B[nj])) (;B[jp];W[kq];B[iq];W[ir];B[hr];W[nq];B[mq];W[jo];B[kn];W[ip] ;B[jq];W[hq];B[jr];W[kr];B[is];W[nr];B[mr];W[ms];B[ls];W[ks] ))