Clarify blob handling.

[Julian-O]

In Python, the only semantic difference between bytes and bytearray data structures, is that the former are immutable and the latter are mutable. It is similar to the difference between tuples and lists.

However, in oscpy, bytes are treated as as an OSC-String and bytesarrays are treated as an OSC-Blob. They are treated differently as to whether they go through an encoding/decoding stage,

This distinction is not obvious nor intuitive.

Please consider treating bytes as OSC-Blobs, to make then consistent. Unicode strings should continue to be treated as strings and encoded/decoded.

Failing that please consider documenting this in the Gotcha section.

[tshirtman]

That's a pretty important change. OSCPy started as only supporting encoded strings (bytes) as input for strings, before adding the encoding parameter, which allows to send unicode strings and automatically get encoding/decoding at both ends, but the default, when no encoding is passed, is to only accept bytes. The usage of bytearray is a bit of a semantical hack to send blobs, as because there was no way to distinguish between strings and blobs in that situation, casting to bytearray made some sense, as it's often used to manage binary data. I'm open to finding a better way, but the proposed change feels too radical to me, and would certainly break existing applications.

[Julian-O]

Yeah, I get that oscpy was written with Py2.7 in mind, and the bad old days where we didn't properly distinguish between characters and bytes.

At the very least, I think this should be clearly documented. Blob handling is barely mentioned, and I had to dig into code and the OSC protocol to work out why my data was being mangled.

If you are not willing to break code in the beta stage, consider providing a helper class to allow the client to tag the data before it is sent.

My first thought was a subclass of bytes called OscBlob. You could treat it very much like a bytes object, but when it was passed to send_message as a values parameter, it would be detected as a blob. Perhaps also a bytearray version too, for consistency, even though it isn't strictly required.

But it is likely the client code would use bytes throughout, and only tag it just before it gets sent. The conversion from a bytes object would involve a copy. The object might be on the order of 64KB (or whatever the UDP packet size limits are for your network). Some devices might see that as a performance hit.

So, I also considered a wrapper class that takes either of bytes or bytesarray instance as a param and just keeps a reference. Again, it doesn't do much, but when it is send to send_message, it is classified as a blob.

Alternatively, be bold. You are still in beta, you can get away with it! :-) If I understand it correctly, it only affects users who want to send text, but choose to send bytes rather than Unicode and who also specify an encoding, which is an odd combo. Make the change to the API to distinguish between bytes and text, and bring it up to the expectations of software in the early 2010s! :-)

[systemeFriche]

Hi, I want to imitate an OSC server (mixing table). My OSC server must reponds 27 float values as an OSC blob. But I don't know how to send a blob.

This is the OSC packet, I have to reproduce :

I write this code :

    X32_ADDRESS = '192.168.1.10'
    osc = OSCThreadServer(encoding='utf8')
    osc.listen(address=X32_ADDRESS, port=10023, default=True)

    @osc.address('/renew')
    def renew(*values):
        value = values[0]
        if value == "meters/5":
            print(value)
            meters = [0.5, 0.4, 0.3, 0.2, 0.1, 0.0, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0, 0.5, 0.4, 0.3, 0.2, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0]
            osc.answer('meters/5', meters)

This code produces this OSC packet :

This is not quite the same. Thank you very much for helping !!!!

[tshirtman]

As previously discussed (and really sorry for never properly addressing it), currently "bytearray" is used to create blobs, so you can simply cast your list as a bytearray.

meters = bytearray([0x6f, 0xbb, ...]) Please note that bytearray is meant to hold bytes so whole numbers between 0 and 255 (or as hex, between 0x0 and 0xff), not floats.

I agree it’s not an ideal api design, and it might change in the future, but that’s how it works with the current version of oscpy.

[systemeFriche]

Thanks to your quick answer, I modify my code like this :

    X32_ADDRESS = '192.168.1.10'
    osc = OSCThreadServer(encoding='utf8')
    osc.listen(address=X32_ADDRESS, port=10023, default=True)

    @osc.address('/renew')
    def renew(*values):
        value = values[0]
        if value == "meters/5":
            print(value)
            meter1 = [0x00, 0x00, 0x00, 0x70]
            meter2 = [0x1b, 0x00, 0x00, 0x00]
            meter = [0x6f, 0xbb, 0xd5, 0x34]
            meters_array = meter1 + meter2
            for _ in range(27):
                meters_array += meter
            meters = bytearray(meters_array)
            osc.answer('meters/5', meters)

I try to reproduce the sniffed OSC packet. The result is :

So I don't have a blob but several integers and I don't know why, but each byte of my byterarray produces a four bytes number in my OSC packet. Very strange. Is it because my OSCThreadServer encoding is utf-8.

Any Idea ? Big thanks.

[systemeFriche]

I tried without utf8 encoding like this :

    X32_ADDRESS = '127.0.0.1'
    X32_ADDRESS_BYTE = b'127.0.0.1'
    osc = OSCThreadServer()
    osc.listen(address=X32_ADDRESS, port=10023, default=True)

    @osc.address(b'/renew')
    def renew(*values):
        value = values[0]
        # array = value.split("/")
        if value == b'meters/5':
            meter1 = [0x00, 0x00, 0x00, 0x70]
            meter2 = [0x1b, 0x00, 0x00, 0x00]
            meter = [0x6f, 0xbb, 0xd5, 0x34]
            meters_array = meter1 + meter2
            for _ in range(27):
                meters_array += meter
            meters = bytearray(meters_array)
            osc.answer(b'meters/5', meters)

but it gives the same result. the osc.answer method considers my bytearray like an array of integers and then each integer is coded on 32 bits.

[tshirtman]

because you send it as the full list of message parameters, so each item is considered a separate item, you need to put meters itself in a list or tuple osc.answer(b"meters/5’, [meters]).

[systemeFriche]

I tried :

            meter1 = [0x00, 0x00, 0x00, 0x70]
            meter2 = [0x1b, 0x00, 0x00, 0x00]
            meter = [0x6f, 0xbb, 0xd5, 0x34]
            meters_array = meter1 + meter2
            for _ in range(27):
                meters_array += meter
            meters = bytearray(meters_array)
            osc.answer('meters/5', (meters,))

and

osc.answer('meters/5', [meters])

in both case i 've got the same python error :

line 271, in format_message message = pack( struct.error: pack expected 122 items for packing (got 3)

[felixdollack]

I have the same problem. Originally, I wanted to receive a blob message, but the blob size parameter is somehow misinterpreted (see #72)? When I invert my goal and instead try to create the message I am trying to receive I also get the struct.error.

Create an OSC message with a blob of size 8:

from oscpy import parser

msg, stats = parser.format_message(
  b'/destn/1', [bytearray([0x01, 0x23, 0x45, 0x67, 0x89, 0xAB, 0xCD, 0xEF])]
)
print(msg, stats)

Error message:

Traceback (most recent call last):
  File "oscpy_send.py", line 3, in <module>
    msg, stats = parser.format_message(
  File "miniconda3/envs/v/lib/python3.9/site-packages/oscpy/parser.py", line 271, in format_message
    message = pack(
struct.error: pack expected 18 items for packing (got 3)

[felixdollack]

@systemeFriche my pull request to receive blob messages just got merged into master. If you try what you did using the latest commits again I think it might work.

I tried :

            meter1 = [0x00, 0x00, 0x00, 0x70]
            meter2 = [0x1b, 0x00, 0x00, 0x00]
            meter = [0x6f, 0xbb, 0xd5, 0x34]
            meters_array = meter1 + meter2
            for _ in range(27):
                meters_array += meter
            meters = bytearray(meters_array)
            osc.answer('meters/5', (meters,))

and

osc.answer('meters/5', [meters])

in both case i 've got the same python error :

line 271, in format_message message = pack( struct.error: pack expected 122 items for packing (got 3)

[systemeFriche]

@felixdollack thank you for your PR but I've got the same python error with

osc.answer('meters/5', (meters,))

and with

osc.answer('meters/5', [meters])

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/oscpy-0.6.1.dev0-py3.10.egg/oscpy/parser.py", line 271, in format_message message = pack( struct.error: pack expected 114 items for packing (got 3)

[felixdollack]

Ah, sorry I misread the error message. You have the problem on the packing side, not the unpacking 🙈 I have no good idea yet how to fix the packing side of things. @tshirtman could it be working by introducing an optional parameter to specify the format and overwrite the internal generated one?

Something like this to send a binary blob:

osc.answer('meters/5', (meters,), fmt='b')