kaitai_struct_compiler icon indicating copy to clipboard operation
kaitai_struct_compiler copied to clipboard

Python/Construct: Use `unsigned char` representation when converting to `bytes`

Open Mingun opened this issue 1 year ago • 0 comments

Currently Python (and Construct) KST test tests\formats\expr_bytes_non_literal.ksy

meta:
  id: expr_bytes_non_literal
seq:
  - id: one
    type: u1
  - id: two
    type: u1
instances:
  calc_bytes:
    value: '[one, two].as<bytes>'

failed with the following error:

Traceback (most recent call last):
  File "/tests/spec/construct/test_expr_bytes_non_literal.py", line 9, in test_expr_bytes_non_literal
    r = _schema.parse_file('src/enum_negative.bin')
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 309, in parse_file
    return self.parse_stream(f, **contextkw)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 300, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 312, in _parsereport
    obj = self._parse(stream, context, path)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 1982, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 312, in _parsereport
    obj = self._parse(stream, context, path)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 2440, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 312, in _parsereport
    obj = self._parse(stream, context, path)
  File "/usr/local/lib/python2.7/site-packages/construct/core.py", line 2576, in _parse
    return self.func(context) if callable(self.func) else self.func
  File "/tests/compiled/construct/expr_bytes_non_literal.py", line 8, in <lambda>
    'calc_bytes' / Computed(lambda this: struct.pack('2b', this.one, this.two)),
error: byte format requires -128 <= number <= 127

The reason is because PythonTranslator uses signed char representation of bytes when casting to bytes type using struct.pack('Nb'), but it tries to pack from u1 (unsigned) integers. So 0xFF from test data raises an error.

It is unclear, what to do here. I speculate, that we expect that arrays for conversion must contain integers in the range [0; 255] and implement it in my PR, but that is only my assumption. I suspect is it correct, although, because both Perl and Ruby uses 'C*' which in both means unsigned chars.

Also, I think, we should add tests with s1 types, alone and mixed with u1 types. What the expected result of converting arrays of such values to bytes type?

Mingun avatar Mar 18 '24 16:03 Mingun