Bindings: setExternal crashes for RGB images with python>=3.6

[Nicogene]

With python>=3.6 this script crashes:

import yarp
import numpy as np

buff_array = np.ones((height, width, 3), dtype=np.uint8)
yarpImageRGB = yarp.ImageRgb()
yarpImageRGB.resize(width, height)
yarpImageRGB.setExternal(buff_array, width, height) # <- Segmentation fault

With @Arya07 we tried the same script with python 3.5 and 2.7 and it works smoothly. The numpy version we tried was 1.15.2

[David-Estevez]

Same thing here with:

• Ubuntu Bionic current default python3: Python 3.6.5 (default, Apr 1 2018, 05:46:30) [GCC 7.3.0] on linux
• Ubuntu Bionic current default python3-numpy: 1:1.13.3-2ubuntu1 (shows as 1.13.3 via pip list)

[pattacini]

@Arya07 is there any possible way to get around this for the time being in order to avoid using setExternal?

For example, in C++ I don't use an external buffer to let the yarp image wrap over.

[dcam0050]

I have seen the same behaviour with setExternal.

I tried the following:

yarp_image = yarp.ImageRgb()
image_bytes = data.astype(dtype=np.uint8).tobytes()
#Convert each byte into a character and join. Using map to make it faster than list comprehension
image_string = "".join(map(chr, image_bytes))
yarp_image.fromstring(image_string, data.shape[0], data.shape[1])

I then immediately try to extract the image as follows: received_strimage = yarp_image.tostring()

and this is the error I get:

Traceback (most recent call last):
  File "/home/icub/user_files/pycharm-2018.2.1/helpers/pydev/_pydevd_bundle/pydevd_xml.py", line 264, in frame_vars_to_xml
    xml += var_to_xml(v, str(k), evaluate_full_value=eval_full_val)
  File "/home/icub/user_files/pycharm-2018.2.1/helpers/pydev/_pydevd_bundle/pydevd_xml.py", line 357, in var_to_xml
    xml_value = ' value="%s"' % (make_valid_xml_value(quote(value, '/>_= ')))
  File "/usr/lib/python3.6/urllib/parse.py", line 783, in quote
    string = string.encode(encoding, errors)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcec' in position 21: surrogates not allowed
Unexpected error, recovered safely.

@pattacini Wrapping around a numpy array is the fastest (and most convenient) way of doing it in python compared to other available approaches

[BrutusTT]

Running into the same problem. (Python 3.6.5, numpy 1.14.0, OSX 10.13.6) Is a solution or workaround available yet?

[Nicogene]

From my understanding I don't have clear whose is fault. Here the possible suspects:

• yarp
• swig
• python
• numpy

The main problem is to debug it, anyone knows tools/technique to debug python scripts?

[BrutusTT]

A minimal test would be:

import numpy as np
import yarp

i = yarp.ImageRgb()
a = np.zeros( (10,10,3), np.uint8)
i.setExternal(a, 10, 10)

This will trigger the segfault. As far as I can see the error occurs within the call of the C++ function. And since it is about a buffer that gets cast to a char array down the line, I suspect that the memory layout in Python might have changed. The only thing I could spot is that Python 3.6 moved from "malloc" to "pymalloc" for memory allocation. Not sure if this relevant, tho.

I suspect the line 188 of Image.cpp causing the segfault (haven't had the time to tested it yet):

pImage->imageData = (char*)buf;

If this is the case it would prove that a change in Python would cause it as this is quite old code that didn't changed and worked previously in the same setup.

[Nicogene]

Thank you @BrutusTT for the analysis, if you can dig into it and find some information about the changes in the memory layout in Python it would be very helpful!

[BrutusTT]

It was mentioned in here: https://docs.python.org/3/c-api/memory.html

Changed in version 3.6: The default allocator is now pymalloc instead of system malloc().

As I said, I am not sure if this is actually causing the trouble but it was the only related thing I saw on the changelog for Python3.6 that might have an impact.

[BrutusTT]

Slightly more info ...

For some unknown reason the call of ::setExternal from the SWIG wrapping (yarp.i:1234) segfaults without even entering the method on the C++ side. This happens with Python3.6 while with Python2.7 the method is entered (meaning a printf put in the first line of the method gets executed with Python2.7 only).

The identical SWIG code was generated for both Python versions as well as the rest of the system kept identical as well. So it is definitely only depending on the Python version ... but I do not see why as the processing does not even enter the function.

That's as far as I can get at the moment. Sadly, no GDB due to Docker which would be the next step I see here.

[Nicogene]

This is the code generated by SWIG:

void setExternal(yarp::sig::Image *img, PyObject* mem, int w, int h) {
#if PY_VERSION_HEX >= 0x02070000
        Py_buffer img_buffer;
        int reply;
        reply = PyObject_GetBuffer(mem, &img_buffer, PyBUF_SIMPLE); // <-- SEGFAULT
        // exit if the buffer could not be read
        if (reply != 0)
        {
            fprintf(stderr, "Could not read Python buffers: error %d\n", reply);
            return;
        }
        img->setExternal((void*)img_buffer.buf, w, h);
        // release the Python buffers
        PyBuffer_Release(&img_buffer);
#else
        fprintf(stderr, "Your python version is not supported\n");
#endif
}

Debugging with gdb, it crashes when PyObject_GetBuffer is called. I have no idea how it is implemented, but maybe it changed. https://docs.python.org/3/c-api/buffer.html

[BrutusTT]

The function is defined in https://github.com/python/cpython/blob/v3.6.5/Objects/abstract.c line 320 and following. Line 331 seems the one where it breaks. Don't know much about the stuff that happens there and it also seems to have not changed from 3.5 to 3.6 (only earlier on).

[BrutusTT]

If one changes the test example to use a bytearray type the segfault does not happen:

import yarp
i = yarp.ImageRgb()
a = bytearray([1,2,3,4])
i.setExternal(a, 2, 2)

From my understanding of Pythons buffer implementation *pb->bf_getbuffer is a function implemented by the datatype in our original case numpy.ndarray. In theory, only a bytes-like object is required for the setExternal function and might work as long as the memory layout fits. Hence, the bytearray object seems to run through.

The whole buffer implementation changed in Python 3.3 with the introduction of PEP-3118. There seems to be an issue with Numpy and PEP-3118 (https://github.com/numpy/numpy/issues/9456) which might explain the difference in behavior between Python 2 and 3 but I am a little bit puzzled why it works on Python 3.5 as the changes should already be in since two minor versions. Nonetheless, the cause of the problem seems now to be the ndarray implementation specifically the buffer handling (not sure if I am motivated enough to take a closer look tho).

...

Anyways, as a workaround I got the image transport working using the bytearray and numpy's frombuffer and tobytes methods. However, that adds roughly 10ms to the processing per image which is not ideal in my case and probably an insane workaround for a method that is meant to be memory and time efficient.

[andylucny]

Thank you for perfect advices, just to summarize the solution:

import yarp import numpy as np import matplotlib.pylab as plt

yarp.Network.init()

port_in = yarp.Port() in_port_name = '/test/cam/left' print(port_in.open(in_port_name)) yarp.delay(0.5) yarp.Network.connect('/icubSim/cam/left','/test/cam/left')

yarp_img_in = yarp.ImageRgb() yarp_img_in.resize(320,240) img_array = bytearray(230400) yarp_img_in.setExternal(img_array, 320, 240) print(port_in.read(yarp_img_in)) print(240,' == ',yarp_img_in.height()) print(320,' == ',yarp_img_in.width()) img = np.frombuffer(img_array, dtype=np.uint8).reshape(240,320,3)

plt.imshow(img) plt.show()

[tfietzek]

I had the same problem. I found another solution with direct usage of a numpy array:

yarp_img = yarp.ImageRgb() yarp_img.resize(320,240) img_array = np.ones((240, 320, 3), np.uint8) yarp_img.setExternal(img_array.data, img_array.shape[1], img_array.shape[1])

This works for me without SegmentationFault (Python3.7; Numpy 1.16; Yarp 3.2)

[vtikha]

@torstenfol thank you for the suggestion. It looks reasonable and will try it out on our systems. Will let you know.

[tfietzek]

I had the same problem. I found another solution with direct usage of a numpy array:

yarp_img = yarp.ImageRgb() yarp_img.resize(320,240) img_array = np.ones((240, 320, 3), np.uint8) yarp_img.setExternal(img_array.data, img_array.shape[1], img_array.shape[1]) this should of course be: yarp_img.setExternal(img_array.data, img_array.shape[1], img_array.shape[0]) Sorry, for the mistake.

[aliomair080369]

Hi all, I am using Ubuntu 18.04.5 LTS (Bionic Beaver), python v3.6.9, Numpy v1.19.5, Yarp v3.3 and Opencv v4.2.0.

I wrote a script to read an image from the folder using opencv and wrote that image to a port '/sender'. I wrote another script that listens to the port '/sender' and reads that image from that port and displays it using plt.imshow().

Since I had faced the similar problems with setExternal() as mentioned in the posts above so I tried to follow the tobytes() workaround to transport the image between two ports.

When I send the image from /sender to /reader, the only thing I get back is a black image. I don't know where I made the mistake. Can someone please have a look at my scripts (below) and point out my mistake.

@andylucny @BrutusTT @Nicogene @drdanz

################################################################################################ Here is the script of sender.py. It reads the image from the folder and opens and writes to /sender port:

import numpy as np
import yarp
import cv2
import matplotlib.pyplot as plt

yarp.Network.init()  #network initializing

image_path = 'bottles_01.jpg'  # image to load from the folder

cap = cv2.VideoCapture(image_path,cv2.CAP_IMAGES) #using cv2.VideoCapture since later I want to read frames from camera
_, frame = cap.read() 

height, width, channels = frame.shape  # height = 230, width = 219, channels = 3

i = yarp.ImageRgb()  # yarp image
i.resize(width,height) # yarp image resizing

a = frame.tobytes()  
i.setExternal(a, 219, 230)

#img = np.frombuffer(a, dtype=frame.dtype).reshape(230,219,3)
#plt.imshow(img)
#plt.show()

output_port = yarp.Port() # initializing the port
output_port.open("/sender") # opening the port /sender

yarp.Network.connect('/sender' , '/reader') # connecting the /sender port with /reader port
output_port.write(i) # writing the image to the port

output_port.close(); # cleaning up
yarp.Network.fini();

######################################################################################## Here is the script of reader.py. It opens a /reader port which reads the image from the /sender and displays it using plt.imshow():

import yarp
import numpy as np
import matplotlib.pylab as plt

yarp.Network.init()# network initializing

port_in = yarp.Port() # initializing the port
port_in.open('/reader') # opening the /reader port 


i = yarp.ImageRgb() # yarp image 
i.resize(219,230)

a = np.zeros((230,219,3)) # image buffer to store the image
b = a.tobytes()

i.setExternal(b,219,230) # wraping the yarp image around the image buffer

port_in.read(i) # reading the image
img = np.frombuffer(b).reshape(230,219,3) # getting the image from the buffer

plt.imshow(img)
plt.show()

port_in.close();
yarp.Network.fini();

################################################################################################### On terminal 1, I start yarpserver. On terminal 2, i run 'python3 reader.py' On terminal 3, i run 'python3 sender.py'

here is the image that I am trying to transport

And here is what I get at the end

[aliomair080369]

I did further digging and found that my image is being transported between two ports. I checked that by saving the received image via yarp.write(i, 'name_of_the_file').

It means that the problem arises when I try to convert the yarp image into python image by using 'img = np.frombuffer(b).reshape(230,219,3)'. But i don't know why it is like that. Any idea?

Is there any way to access the pixel values of the yarp image that I received by 'port_in.read(i)' ?

I saw that in yarp.i, there is a way to get the pixel values of ImageFloat by getPixel() method but I couldn't find a way to get the pixel values of for ImageRgb. Or did i miss something?

Any help would be highly appreciated.

Thanks in advance.

[tfietzek]

Hello @jazzscout, in testing I found as a first problem, that the array a and the function call frombuffer(b) in the reader.py script are not typed. This should be: a = np.zeros((230, 219, 3), dtype=np.uint8) np.frombuffer(b, dtype=np.uint8).reshape(230, 219, 3)

In this way I can compare the array with a cv-loaded version elementwise, hich shows that they are identical, but I run into in error with plotting them with matplotlib, which I couldn't find the reason for yet. (python3: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.)

This error also occurs, if I only copy the yarp image in the same file. I will investigate this further and tell you, if I find a solution.

[PeterBowman]

There is a snippet by @jgvictores that used to work without additional numpy operations (no tobytes nor frombuffer): ref (keep in mind this is a depth frame, i.e. using float-type pixels). Relevant chunk of code:

# Create numpy array to receive the image and the YARP image wrapped around it
img_array = numpy.zeros((height, width), dtype=numpy.float32)
yarp_image = yarp.ImageFloat()
yarp_image.resize(width, height)
yarp_image.setExternal(img_array, img_array.shape[1], img_array.shape[0])

# Read the data from the port into the image
input_port.read(yarp_image)

# display the image that has been read
matplotlib.pylab.imshow(img_array,interpolation='none')
matplotlib.pylab.show()

[tfietzek]

This Code does not work anymore for newer Python versions. At least for me it works with a small modification by replacing: yarp_image.setExternal(img_array, img_array.shape[1], img_array.shape[0]) with: yarp_image.setExternal(img_array.data, img_array.shape[1], img_array.shape[0])

Anything else can then stay the same and as you told no additional numpy action is necessary.

[aliomair080369]

Hi @torsten93, thank you for the support. I also get the same error now with plotting them with matplotlib as you mentioned above.

I also tried using your solution 'yarp_image.setExternal(img_array.data, img_array.shape[1], img_array.shape[0])'. With this also, i get the same error with plotting them with matplotlib.

As you said that the code snippet mentioned by @PeterBowman does not work anymore with the newer Python versions, could you please tell me which python version should I use to get this work? I have python 3.6.9 as of yet.

Thanks in advance.

[jgvictores]

I'm also confirming a segfault on Python 3.8.5 with the same code (setExternal call), even downgrading to numpy 1.15.0. Out of curiosity, I ran a gdb --args python3 exampleDepthImageRead.py, here's the bt (EDIT: I hadn't read https://github.com/robotology/yarp/issues/1873#issuecomment-771601856 thanks!):

(gdb) bt
#0  0x0000000000558ee6 in PyErr_Clear ()
#1  0x00000000005d3576 in PyDict_GetItem ()
#2  0x00007ffff6a327f9 in _buffer_get_info (obj=obj@entry=0x7ffff6c26080)
    at numpy/core/src/multiarray/buffer.c:584
#3  0x00007ffff6a32b5b in array_getbuffer (obj=0x7ffff6c26080, view=0x7fffffffcf40, flags=0)
    at numpy/core/src/multiarray/buffer.c:713
#4  0x00007ffff391244b in _wrap_Image_setExternal ()
    at /usr/local/lib/python3.8/dist-packages/_yarp.so
#5  0x00000000005f42ea in PyCFunction_Call ()
#6  0x0000000000570f85 in _PyEval_EvalFrameDefault ()
#7  0x000000000056955a in _PyEval_EvalCodeWithName ()
#8  0x00000000005f7323 in _PyFunction_Vectorcall ()
#9  0x000000000056b399 in _PyEval_EvalFrameDefault ()
#10 0x000000000056955a in _PyEval_EvalCodeWithName ()
#11 0x000000000068c4a7 in PyEval_EvalCode ()
#12 0x000000000067bc91 in  ()
#13 0x000000000067bd0f in  ()
#14 0x000000000067bdcb in PyRun_FileExFlags ()
#15 0x000000000067de4e in PyRun_SimpleFileExFlags ()
#16 0x00000000006b6032 in Py_RunMain ()
#17 0x00000000006b63bd in Py_BytesMain ()
#18 0x00007ffff7dbc0b3 in __libc_start_main (main=

[aliomair080369]

Hi @torsten93,

I just wanted to update you regarding that problem. I apparently found a way of transporting and plotting the images. But I don't know why and how it actually works. I followed your solution to transport the images between two ports.

yarp_image.setExternal(img_array.data, img_array.shape[1], img_array.shape[0])

But before transporting the image, I resized it using cv2.resize() function as shown in code below (sender.py). The strange thing is that if I resize the image to lets say (width = 219 and height = 215), it shows the same error that you mentioned while plotting the received image by plt.imshow() in reader.py

However, it works well when I resize the image to lets say (width = 320 and height = 240) or (width = 416 and height = 416) etc. I don't know why it works well with only certain dimensions of the image. Could it be that because of certain sizes of image that it runs out of memory???

But then if it works for (416x416x3) then it definitely should work for (230x219x3) which is the original dimension of the loaded image. This is strange and I couldn't figure out the reason for that.

here is the sender.py script

import numpy
import yarp
import cv2
import matplotlib.pylab as plt
# Initialise YARP
yarp.Network.init()

# read image from the folder using opencv2  
image_path = 'bottleshapes_01.jpg'
cap = cv2.VideoCapture(image_path,cv2.CAP_IMAGES)
_, img_array = cap.read() # original width = 219, original height = 230

# cv2.resize(img,(width, heigth))
img_array = cv2.resize(img_array,(320,240)) # width = 320, height = 240
#img_array = cv2.resize(img_array,(416,416))320

yarp_image = yarp.ImageRgb()
yarp_image.setExternal(img_array.data, img_array.shape[1], img_array.shape[0])
# Create the yarp port, connect it to the running instance of yarpview and send the image
output_port = yarp.Port()
output_port.open("/sender")
yarp.Network.connect("/sender", "/reader")
output_port.write(yarp_image)
#print(img_array.shape)
yarp.write(yarp_image, 'sender_jpg')
plt.imshow(img_array)
plt.show()

# Cleanup
output_port.close()

here is the script for reader.py:

import numpy
import yarp
import matplotlib.pylab as plt
# Initialise YARP
yarp.Network.init()
# Create a port and connect it to the iCub simulator virtual camera
input_port = yarp.Port()
input_port.open("/reader")
yarp.delay(0.5)

#yarp.Network.connect("/sender", "/reader")
# Create numpy array to receive the image and the YARP image wrapped around it

#numpy.ones(((heigth, width, channels)))
img_array = numpy.ones((240, 320,3), dtype=numpy.uint8)  
#img_array = numpy.ones((416, 416,3), dtype=numpy.uint8)
yarp_image = yarp.ImageRgb()
yarp_image.resize(img_array.shape[1], img_array.shape[0])
yarp_image.setExternal(img_array.data, img_array.shape[1], img_array.shape[0])
# Read the data from the port into the image
input_port.read(yarp_image)

print(img_array)
#print(img_array.shape)
#yarp.write(yarp_image, 'receiver_jpg')
# display the image that has been read
plt.imshow(img_array)
plt.show()
# Cleanup
input_port.close()