simplified prompt gets better moves

[dchudz]

Not sure if you're interested in this sort of issue. Feel free to close, of course.

I found that as written, this makes some pretty silly moves. E.g.:

python main.py
r n b q k b n r
p p p p p p p p
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
P P P P P P P P
R N B Q K B N R
[White "User"]
[Black "ChatGPT"]
[Result "*"]

*
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
['Nh3', 'Nf3', 'Nc3', 'Na3', 'h3', 'g3', 'f3', 'e3', 'd3', 'c3', 'b3', 'a3', 'h4', 'g4', 'f4', 'e4', 'd4', 'c4', 'b4', 'a4']
Enter your move: e4
ChatGPT played Nf6 because ...Nf6 allows black to develop a piece and control the central d5 square. This move also sets up potential threats to White's pawn on e4, such as playing d5 later on. Additionally, by developing the knight to its natural square, Black prepares to castle kingside and further mobilize their pieces for an attack.
r n b q k b . r
p p p p p p p p
. . . . . n . .
. . . . . . . .
. . . . P . . .
. . . . . . . .
P P P P . P P P
R N B Q K B N R
[White "User"]
[Black "ChatGPT"]
[Result "*"]

1. e4 Nf6 *
rnbqkb1r/pppppppp/5n2/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - 1 2
['Nh3', 'Nf3', 'Ne2', 'Ba6', 'Bb5', 'Bc4', 'Bd3', 'Be2', 'Ke2', 'Qh5', 'Qg4', 'Qf3', 'Qe2', 'Nc3', 'Na3', 'e5', 'h3', 'g3', 'f3', 'd3', 'c3', 'b3', 'a3', 'h4', 'g4', 'f4', 'd4', 'c4', 'b4', 'a4']
Enter your move: Nc3
ChatGPT played Nc6 because ...Nc6 continues Black's development while attacking White's undefended pawn on d4. This move is a common response to White's Nc3 move and puts pressure on White to make a decision on how to defend the pawn or give it up. Additionally, this move opens up the possibility of a future d5 break in the center, further challenging White's space advantage.
r . b q k b . r
p p p p p p p p
. . n . . n . .
. . . . . . . .
. . . . P . . .
. . N . . . . .
P P P P . P P P
R . B Q K B N R
[White "User"]
[Black "ChatGPT"]
[Result "*"]

1. e4 Nf6 2. Nc3 Nc6 *
r1bqkb1r/pppppppp/2n2n2/8/4P3/2N5/PPPP1PPP/R1BQKBNR w KQkq - 3 3
['Nd5', 'Nb5', 'Na4', 'Nce2', 'Nb1', 'Nh3', 'Nf3', 'Nge2', 'Ba6', 'Bb5', 'Bc4', 'Bd3', 'Be2', 'Ke2', 'Qh5', 'Qg4', 'Qf3', 'Qe2', 'Rb1', 'e5', 'h3', 'g3', 'f3', 'd3', 'b3', 'a3', 'h4', 'g4', 'f4', 'd4', 'b4', 'a4']
Enter your move: d4
ChatGPT played Nd5 because ...Nd5 attacks White's pawn on c3 and forces it to move, which weakens White's control over the d4 square. Black can then potentially occupy that square with a pawn or a knight in the future. Additionally, Nd5 develops a piece and prepares to castle kingside. Overall, Nd5 improves Black's position and puts pressure on White.
r . b q k b . r
p p p p p p p p
. . n . . . . .
. . . n . . . .
. . . P P . . .
. . N . . . . .
P P P . . P P P
R . B Q K B N R
[White "User"]
[Black "ChatGPT"]
[Result "*"]

1. e4 Nf6 2. Nc3 Nc6 3. d4 Nd5 *
r1bqkb1r/pppppppp/2n5/3n4/3PP3/2N5/PPP2PPP/R1BQKBNR w KQkq - 1 4
['Nxd5', 'Nb5', 'Na4', 'Nce2', 'Nb1', 'Nh3', 'Nf3', 'Nge2', 'Ba6', 'Bb5', 'Bc4', 'Bd3', 'Be2', 'Ke2', 'Kd2', 'Qh5', 'Qg4', 'Qf3', 'Qd3', 'Qe2', 'Qd2', 'Bh6', 'Bg5', 'Bf4', 'Be3', 'Bd2', 'Rb1', 'exd5', 'e5', 'h3', 'g3', 'f3', 'b3', 'a3', 'h4', 'g4', 'f4', 'b4', 'a4']
Enter your move: exd5
ChatGPT played Na5 because he knight move to a5 attacks the c4 square and prepares to put pressure on the white pawn on b3. This move also allows for the development of the bishop on c8, which can be brought out to b7 or a6. Additionally, the knight on a5 can also potentially control the b3 square and limit White's pawn breaks on the queenside. Overall, this move helps to increase Black's control over the center and put pressure on White's position.
r . b q k b . r
p p p p p p p p
. . . . . . . .
n . . P . . . .
. . . P . . . .
. . N . . . . .
P P P . . P P P
R . B Q K B N R
[White "User"]
[Black "ChatGPT"]
[Result "*"]

By simplifying the prompt, I was able to get much better moves:

def generate_prompt(game: chess.pgn.Game, board: chess.Board) -> str:
    """Generates a prompt for OpenAI."""
    moves = get_legal_moves(board)
    moves_str = ",".join(moves)
    return f"""{str(game.mainline_moves())} ...

Please make the next move and restate the game history in the same format.
"""

def get_move_from_prompt(prompt: str) -> tuple[str,str]:
    """Returns the move from the prompt."""
    response = openai.ChatCompletion.create(
      model="gpt-3.5-turbo",
      messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt},
        ]
    )
    content = response['choices'][0]['message']['content']
    print(content)
    mv = content.split()[-1].strip(".")

Presumably my version needs a bit more error checking for bad parses on the move, but it has worked for me so far.