chess-tuning-tools icon indicating copy to clipboard operation
chess-tuning-tools copied to clipboard

Add more validation of cutechess output

Open thomasahle opened this issue 1 year ago • 1 comments

  • Chess Tuning Tools version: 0.9.3
  • Python version: 3.9
  • Operating System: Linux

Description

I had some really strange errors (see trace below) which turned out to be because my engine would sometimes (in branches it didn't expect to happen) write some stuff to stdout. This would mess with the regular expression matching the cutechess-cli results.

Instead of the cryptical "all the input arrays must have same number of dimensions" error, it would be nice with something like "Unexpected output: {...}"

What I Did

2023-01-06 08:50:36,242 INFO     Start experiment
2023-01-06 08:53:08,948 INFO     Experiment finished (152.704942s elapsed).
Traceback (most recent call last):
  File "/fsx/users/ahle/conda/bin/tune", line 8, in <module>
    sys.exit(cli())
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/cli.py", line 505, in local
    score, error_variance, draw_rate = parse_experiment_result(out_exp, **settings)
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/tune/local.py", line 1047, in parse_experi
ment_result
    diffs = np.diff(array, axis=0, prepend=np.array([[0, 0, 0]]))
  File "<__array_function__ internals>", line 5, in diff
  File "/fsx/users/ahle/conda/lib/python3.9/site-packages/numpy/lib/function_base.py", line 1281, in
 diff
    a = np.concatenate(combined, axis)
  File "<__array_function__ internals>", line 5, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2
 dimension(s) and the array at index 1 has 1 dimension(s)
srun: error: dev-dy-p3dn24xlarge-1: task 0: Exited with exit code 1

thomasahle avatar Jan 06 '23 09:01 thomasahle