Autopkgtest failure on big-endian (s390x)

[ginggs]

I see the autokpgtest failure below in Ubuntu 19.10 on s390x, with meshio 3.1.6,

From http://autopkgtest.ubuntu.com/packages/p/pygalmesh/eoan/s390x

___________________________ test_volume_from_surface ___________________________

    def test_volume_from_surface():
        this_dir = os.path.dirname(os.path.abspath(__file__))
        mesh = pygalmesh.generate_volume_mesh_from_surface_mesh(
            os.path.join(this_dir, "meshes", "elephant.vtu"),
            facet_angle=25.0,
            facet_size=0.15,
            facet_distance=0.008,
            cell_radius_edge_ratio=3.0,
>           verbose=False,
        )

test/test_volume_from_surface.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3/dist-packages/pygalmesh/main.py:188: in generate_volume_mesh_from_surface_mesh
    mesh = meshio.read(filename)
/usr/lib/python3/dist-packages/meshio/_helpers.py:192: in read
    return format_to_reader[file_format].read(filename)
/usr/lib/python3/dist-packages/meshio/_vtu.py:321: in read
    reader = VtuReader(filename)
/usr/lib/python3/dist-packages/meshio/_vtu.py:194: in __init__
    pts = self.read_data(data_array)
/usr/lib/python3/dist-packages/meshio/_vtu.py:308: in read_data
    data = self.read_binary(c.text.strip(), c.attrib["type"])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <meshio._vtu.VtuReader object at 0x3ffa2b92948>
data = 'AwAAAACAAAAoBAAAeXEAAA9yAAAzBAAAeJw8XHc81f/3r09GRdGS0k5pSSkl4aBSKZJVWmiIhMhOJSHZe8/svfc4d7ruNa+9ZZSsprSM373q++sPPR7XX...uFv5c5XRYTm8O/FHr9SXzuyj+SBUzg1t2MvzofniXICMS/m44AbhYvyj/JjjKHSV2Xv0cAN4sXC8O/zbBR1m8mvr8hWcAEbt3Jv/Flogip28k/XHIkXw=='
data_type = 'Float64'

    def read_binary(self, data, data_type):
        # first read the the block size; it determines the size of the header
        dtype = vtu_to_numpy_type[self.header_type]
        num_bytes_per_item = numpy.dtype(dtype).itemsize
        num_chars = num_bytes_to_num_base64_chars(num_bytes_per_item)
        byte_string = base64.b64decode(data[:num_chars])[:num_bytes_per_item]
        num_blocks = numpy.frombuffer(byte_string, dtype)[0]
    
        # read the entire header
        num_header_items = 3 + num_blocks
        num_header_bytes = num_bytes_per_item * num_header_items
        num_header_chars = num_bytes_to_num_base64_chars(num_header_bytes)
        byte_string = base64.b64decode(data[:num_header_chars])
>       header = numpy.frombuffer(byte_string, dtype)
E       ValueError: buffer size must be a multiple of element size

/usr/lib/python3/dist-packages/meshio/_vtu.py:270: ValueError

[nschloe]

Hm, interesting. Looks like a rare bug in meshio. Do you have ssh-access to any of those machines?

[ginggs]

I do. Please let me know if there's anything I can test.

[nschloe]

Yep. In test/meshes/ you'll find elephant.vtu. What's the result of

import meshio
meshio.read("elephant.vtu")

on that machine?

[ginggs]

I tried with meshio 3.2.0.

$ python3
Python 3.7.5rc1 (default, Oct  8 2019, 16:47:45) 
[GCC 9.2.1 20190909] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import meshio
>>> meshio.read("elephant.vtu")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ginggs/meshio-3.2.0/meshio/_helpers.py", line 192, in read
    return format_to_reader[file_format].read(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 321, in read
    reader = VtuReader(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 194, in __init__
    pts = self.read_data(data_array)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 308, in read_data
    data = self.read_binary(c.text.strip(), c.attrib["type"])
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 270, in read_binary
    header = numpy.frombuffer(byte_string, dtype)
ValueError: buffer size must be a multiple of element size

[nschloe]

Okay, so this is really a meshio bug. Can you print all of the involved variables plus

print(len(byte_string))

etc?

[ginggs]

For the first block, on s390x:

dtype: uint32
num_bytes_per_item: 4
num_chars: 8
num_blocks: 50331648
num_header_items: 50331651
num_header_bytes: 201326604
num_header_chars: 268435472
len(byte_string): 59347

on amd64:

dtype: uint32
num_bytes_per_item: 4
num_chars: 8
num_blocks: 3
num_header_items: 6
num_header_bytes: 24
num_header_chars: 32
len(byte_string): 24

[nschloe]

Okay, so it's the numblocks. What's the output of

dtype.byteorder
sys.byteorder
byte_string
numpy.frombuffer(byte_string, dtype)

? (Both s390x and amd64 would be nice.)

[ginggs]

s390x:

dtype.byteorder: =
sys.byteorder: big
byte_string: b'\x03\x00\x00\x00'
numpy.frombuffer(byte_string, dtype): [50331648]

amd64:

dtype.byteorder: =
sys.byteorder: little
byte_string: b'\x03\x00\x00\x00'
numpy.frombuffer(byte_string, dtype): [3]

[nschloe]

Possible fix: Add

        if self.byte_order is not None:
            dtype = dtype.newbyteorder(
                "<" if self.byte_order == "LittleEndian" else ">"
            )

after the dtype definition. Could you try that out?

[ginggs]

Now it seems to read the entire file, then:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ginggs/meshio-3.2.0/meshio/_helpers.py", line 192, in read
    return format_to_reader[file_format].read(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 338, in read
    reader = VtuReader(filename)
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 252, in __init__
    point_offsets, cells, cell_data_raw
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 50, in _organize_cells
    cls["connectivity"], cls["offsets"], cls["types"], cdr
  File "/home/ginggs/meshio-3.2.0/meshio/_vtu.py", line 32, in _cells_from_data
    meshio_type = vtk_to_meshio_type[vtk_type]
KeyError: 360287970189639680

[nschloe]

print(vtk_type)

?

[nschloe]

Ah, wait. The error is elsewhere.

[ginggs]

s390x:

vtk_type: 360287970189639680

amd64:

vtk_type: 5

[nschloe]

print(cells)

before the call to _organize_cells (line 250)?

[ginggs]

s390x:

cells: [{'connectivity': array([ 4540191374342881280, -4682617712558473216, -3962041772179193856,
       ...,  1298162592589545472,  7711006986988421120,
       -3383892170015506432]), 'offsets': array([ 216172782113783808,  432345564227567616,  648518346341351424, ...,
       2035908506548174848, 2252081288661958656, 2468254070775742464]), 'types': array([360287970189639680, 360287970189639680, 360287970189639680, ...,
       360287970189639680, 360287970189639680, 360287970189639680])}]

amd64:

cells: [{'connectivity': array([ 575, 1215, 1225, ..., 1042,  875, 2769]), 'offsets': array([    3,     6,     9, ..., 16668, 16671, 16674]), 'types': array([5, 5, 5, ..., 5, 5, 5])}]

[nschloe]

I really don't get it. I think the data is read alright, but I don't understand why the cell data is converted into the wrong endianness now.

Can I get a login on the respective machine or can I send you my public key to temporarily use your account?

[ginggs]

I'm using a Debian porterbox, but you can apply for guest access: https://dsa.debian.org/doc/guest-account/

[nschloe]

Okay, did apply.