RBSManager
RBSManager copied to clipboard
wesgood
Support for sensor_msgs/CompressedImage
is it possible to support sensor_msgs/CompressedImage? http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CompressedImage.html
I can add this message to the library but for now feel free to create a custom message for your specific project. There is a Python script in the repo that can help with it.
I Sent you an merge request. With the Message, was Created with your tool
Mit Freundlichen Grüßen
Simon Helming
On 18. Jan 2019, at 15:07, wesgood [email protected] wrote:
I can add this message to the library but for now feel free to create a custom message for your specific project. There is a Python script in the repo that can help with it.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
I Wrote an merge request nur the Data Array is empty do you have an idea why?
Mit Freundlichen Grüßen
Simon Helming
On 18. Jan 2019, at 15:07, wesgood [email protected] wrote:
I can add this message to the library but for now feel free to create a custom message for your specific project. There is a Python script in the repo that can help with it.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
I haven't tested the CompressedImage message specifically (I'm mainly tied to the turtlesim) but I noticed the format of the nested message types is different than the messages in the pod already. Can you try configuring the values like TwistMessage?
For that I'd suggest these changes:
- property titles are optional
- add init method with nested message types
I wrote but now I got now nil instead of empty array...
import UIKit
import ObjectMapper
import RBSManager
public class CompressedImage: RBSMessage {
public var header: Header?
public var format: String?
public var data: [UInt8]?
public override init() {
super.init()
format = ""
header = Header()
data = [UInt8]()
}
public required init?(map: Map) {
super.init(map: map)
}
public override func mapping(map: Map) {
header <- map["header"]
format <- map["format"]
data <- map["data"]
}
}
You should be able to use breakpoints inside the pod to see what is happening. Perhaps the incoming data doesn't match what is expected. If you can nail down where in the code it gets caught, I might be able to investigate further.
okay, but the change is like what do you want ? isn't it? I will investigate it. the rostopic echo show me the data correct. alright. thanks for help
okay now I debugged it, and got follow result the incoming message to RBSManager was
"{\"topic\": \"/raspicam_node/image/compressed\", \"msg\": {\"header\": {\"stamp\": {\"secs\": 1548186301, \"nsecs\": 302186786}, \"frame_id\": \"raspicam\", \"seq\": 168}, \"data\": \"/9j/4QDeRXhpZgAATU0AKgAAAAgACAEAAAQAAAABAAAFAAEBAAQAAAABAAADwAEQAAIAAAAJAAAAbgEaAAUAAAABAAAAeAEbAAUAAAABAAAAgAEoAAMAAAABAAIAAAITAAMAAAABAAEAAIdpAAQAAAABAAAAiAAAAABCcm9hZGNvbQAAAAAASAAAAAEAAABIAAAAAQAGkAAABwAAAAQwMjIwkQEABwAAAAQBAgMAoAAABwAAAAQwMTAwoAEAAwAAAAEAAQAAoAIABAAAAAEAAAUAoAMABAAAAAEAAAPAAAAAAP/bAIQAAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEB/8AAEQgDwAUAAwEiAAIRAQMRAf/EAaIAAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKCxAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6AQADAQEBAQEBAQEBAAAAAAAAAQIDBAUGBwgJCgsRAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQ"
here you see the rostopic echo /raspicam_node/image/compressed output here it looks like an Array like the spec... so I am confused ... RBSManager tried to transformed into my CommpressedImage Class. compresedImage.txt
header:
seq: 1202
stamp:
secs: 1548188274
nsecs: 559593839
frame_id: "raspicam"
format: "jpg"
data: [255, 216, 255, 225, 0, 222, 69, 120, 105, 102, 0, 0, 77, 77, 0, 42, 0, 0, 0, 8, 0, 8, 1, 0, 0, 4, 0, 0, 0, 1, 0, 0, 5, 0, 1, 1, 0, 4, 0, 0, 0, 1, 0, 0, 3, 192, 1, 16, 0, 2, 0, 0, 0, 9, 0, 0, 0, 110, 1, 26, 0, 5, 0, 0, 0, 1, 0, 0, 0, 120, 1, 27, 0, 5, 0, 0, 0, 1, 0, 0, 0, 128, 1, 40, 0, 3, 0, 0, 0, 1, 0, 2, 0, 0, 2, 19, 0, 3, 0, 0, 0, 1, 0, 1, 0, 0, 135, 105, 0, 4, 0, 0, 0, 1, 0, 0, 0, 136, 0, 0, 0, 0, 66, 114, 111, 97, 100, 99, 111, 109, 0, 0, 0, 0, 0, 72, 0, 0, 0, 1, 0, 0, 0, 72, 0, 0, 0, 1, 0, 6, 144, 0, 0, 7, 0, 0, 0, 4, 48, 50, 50, 48, 145, 1, 0, 7, 0, 0, 0, 4, 1, 2, 3, 0, 160, 0, 0, 7, 0, 0, 0, 4, 48, 49, 48, 48, 160, 1, 0, 3, 0, 0, 0, 1, 0, 1, 0, 0, 160, 2, 0, 4, 0, 0, 0, 1, 0, 0, 5, 0, 160, 3, 0, 4, 0, 0, 0, 1, 0, 0, 3, 192, 0, 0, 0, 0, 255, 219, 0, 132, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,...]
Hmm, it looks to be more of a base64 encoded string, which would make sense for an image, but none of the decoders I've seen are able to do anything with it. Have you seen it working with a different processor so we can reverse engineer it? It might just have to be a different type than UInt8[]
yeah i used the rqt_image_view to show the image and it works perfectly. but I don't understand it, because I used the spec http://docs.ros.org/melodic/api/sensor_msgs/html/msg/CompressedImage.html
Was this resolved? Or dead. I'd really like to use this.
Would something like this "Decoding Base64 image in swift" from stack overflow help? https://stackoverflow.com/questions/51557553/decoding-base64-image-in-swift
Hello! Since the original comment I have learned a little more about the rosbridge array/decoding process and I'd like to look into this more. My problem is that I'm not sure how I can test it. If I plug in a USB webcam, what node or package do you suggest for getting the compressed image message out?
Hello - thanks for your quick response!
I implemented, to the best of my knowledge both compressed image and regular image message types - both needed for my situation. Given that thread that brought me here - my assumption is that the data being transferred in the Compressed case is bse64 encoded. So the code was straightforward. To your last point, don't really know if that works in the ROS environment. Here's a snippet at bottom...
Question, if you don't mind. We set this up in a test environment. We could subscribe fine on the iOS device to simple strings. But every other type of message that we published hit an error. On the iOS side the error message was something like "Starscream WSError 1". However, I'm thinking the error might be on the ROSBridge side. Clearly you have this working fine - but have you ever heard of something like this?
When I have all working I'd enjoy packaging up both the compressed image and image message types. I think it would be useful to also add the methods to get/rcv a UIImage as well as the raw data from the class.
Best! p
encode:
-
let rbcPng = UIImage(named: "rbc.png")!* -
let pngData = rbcPng.pngData()!* -
let pngStr = pngData.base64EncodedString()* -
let message = CompressedImageMessage()* -
message.format = "png"* -
let array: [UInt8] = Array(pngStr.utf8)* -
message.data = array*
decode: (interesting, don't need to use format...)
*let* imgMsg = message *as*! CompressedImageMessage
print("format is \(imgMsg.format)")
*let* encData = Data(imgMsg.data!)
*let* imgData = Data(base64Encoded: encData)
*let* img = UIImage(data: imgData!)
*self*.imgDisplay.image = img
On Mon, Aug 26, 2019 at 10:12 AM wesgood [email protected] wrote:
Hello! Since the original comment I have learned a little more about the rosbridge array/decoding process and I'd like to look into this more. My problem is that I'm not sure how I can test it. If I plug in a USB webcam, what node or package do you suggest for getting the compressed image message out?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW2KFW2ZGSCQZE2G52TQGPQD7A5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5EPWKY#issuecomment-524876587, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THWZ3KJOX7444N3QUYATQGPQD7ANCNFSM4GQ5SZQQ .
In my current robot project I worked out how to receive and load an array of Ints (rosbridge base64 encodes arrays, like you pointed out) so I can try to apply it to the image data.
My question was more about how do you suggest building a node that outputs a compressed image stream? Are you running a webcam + ros node combo that I might be able to replicate?
My images come from a ROS environment on a drone. I'm using the iPad as both a display and as a machine learning platform - the drone doesn't have enough oomph.
If I understand your need, you'd like to publish a video stream from the iOS device, right? (If it's something else, I'd not be a good person to ask.)
I can think of two ways.
- Get your images the 'standard' way you would to show images on the screen. The whole AVCaptureSession, AVCaptureDevice, etc. Googling will give you example code. The challenge is in throttling down from the frame rate you get. You can either set the frame rate (
https://developer.apple.com/documentation/avfoundation/avcapturedevice) or do something like throwing out every other image. (The default is 30fps.)
- Set up your own timing loop and read from the camera. Now that I think of this, I'm not sure how to just 'read from the camera'. So previous is probably best.
On Mon, Aug 26, 2019 at 2:11 PM wesgood [email protected] wrote:
In my current robot project I worked out how to receive and load an array of Ints (rosbridge base64 encodes arrays, like you pointed out) so I can try to apply it to the image data.
My question was more about how do you suggest building a node that outputs a compressed image stream? Are you running a webcam + ros node combo that I might be able to replicate?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW6BKNNB26QMIFE42SDQGQMG5A5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FF4RY#issuecomment-524967495, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THW44GY44XFL4P7D5DTTQGQMG5ANCNFSM4GQ5SZQQ .
Hmm, perhaps I'll grab a USB webcam and experiment. In the meantime try using the user above's message data and see if you can get a valid UIImage out of it. That is probably how I will start my research.
No love. The string itself should be divisible by 4 apparently - and it isn't. Tossing in padding at the end "==" allows a valid base64 decode, but that doesn't parse to a UIImage.
Hmmm. The phrase 'compressed' might be of interest here...
On Mon, Aug 26, 2019 at 2:46 PM wesgood [email protected] wrote:
Hmm, perhaps I'll grab a USB webcam and experiment. In the meantime try using the user above's message data and see if you can get a valid UIImage out of it. That is probably how I will start my research.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW2LT54X5EWMARRDC3DQGQQJDA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FJGPQ#issuecomment-524981054, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THW4NQJEIFZEW3GLYXRDQGQQJDANCNFSM4GQ5SZQQ .
Sounds like a good challenge. The Int message I decoded earlier seemed a little unconventional too so I'll have to look into it further.
The data sure seems to be ascii, i.e., not binary. The obvious thing is base64 decoding. That's hiccup #1 as it doesn't work. Also, this https://answers.ros.org/question/51490/sensor_msgscompressedimage-decompression/ seems to imply that the data should be binary. So confusion #2.
Then that Data could convert to a UIImage. No Dice.
Perhaps it's been compressed somehow. I couldn't decompress - probably my out'o'date swift skills.
Net net, I think I'll back up to standard image message to see if I can get something out of that...
On Mon, Aug 26, 2019 at 3:09 PM wesgood [email protected] wrote:
Sounds like a good challenge. The Int message I decoded earlier seemed a little unconventional too so I'll have to look into it further.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW4K5FQYSHT6XVUL6GDQGQS7JA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FLMYI#issuecomment-524990049, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THW4K5TZ6M4WGHQTO433QGQS7JANCNFSM4GQ5SZQQ .
when you say "couldn't decompress" what did you try? I'm unsure of how I would go about trying to decompress it
started here... https://developer.apple.com/documentation/compression/compressing_and_decompressing_data_with_buffer_compression?language=objc
though now that I think of it, it's doubtful this would be apple related.
probably best to do a gzip decompression...
that, or best yet look at ROS source code!
On Mon, Aug 26, 2019 at 3:30 PM wesgood [email protected] wrote:
when you say "couldn't decompress" what did you try? I'm unsure of how I would go about trying to decompress it
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW6A4ZEZQCUN6P6AEVLQGQVNJA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FNH2Q#issuecomment-524997610, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THWYU33LIAWYNGENYFCLQGQVNJANCNFSM4GQ5SZQQ .
okay, latest - the compression is handled by opencv imencode / imdecode. https://answers.opencv.org/question/188436/what-exactly-does-the-imencode-do-wanted-to-use-it-for-image-upload-onto-server/
Seems clear that the compressed format is binary. (Do ROS messages send anything in binary? Or is it all JSON?)
Sounds like opencv is required then. A pain for RBSManager, if true.
On Mon, Aug 26, 2019 at 3:35 PM Peter Gaston [email protected] wrote:
started here... https://developer.apple.com/documentation/compression/compressing_and_decompressing_data_with_buffer_compression?language=objc
though now that I think of it, it's doubtful this would be apple related.
probably best to do a gzip decompression...
that, or best yet look at ROS source code!
On Mon, Aug 26, 2019 at 3:30 PM wesgood [email protected] wrote:
when you say "couldn't decompress" what did you try? I'm unsure of how I would go about trying to decompress it
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW6A4ZEZQCUN6P6AEVLQGQVNJA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FNH2Q#issuecomment-524997610, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THWYU33LIAWYNGENYFCLQGQVNJANCNFSM4GQ5SZQQ .
In my current robot project I worked out how to receive and load an array of Ints (rosbridge base64 encodes arrays, like you pointed out) so I can try to apply it to the image data.
My question was more about how do you suggest building a node that outputs a compressed image stream? Are you running a webcam + ros node combo that I might be able to replicate?
I used an raspi cam. but I hadn't time to implement it, sorry. my usecase was the raspi cam stream to my ipad, nearly real-time. I hoped I can use this way, because every other way whcih I found was to slow..
I'm having a heck of a time getting compressed image to work. I'm pretty sure from the documentation that opencv imencode/decode are the key functions to call.
However, I'm having trouble getting there from Swift. I've done C++ bridging before, but never with arrays/pointers to arrays. Any thoughts there?
btw, this might help in your speed issue as well, though who knows where your slowup is in your configuration.
p
Code now - doesn't work. Can't get a Swift function call to successfully compile.
C++
-
(UIImage *) decodeToImage:(unsigned char *) buff {
cv::Mat mat = cv::imdecode( (cv::InputArray) buff, -1);
if ( mat.empty() ) {
*return* *nil*;}
cv::cvtColor(mat, mat, cv::COLOR_BGR2RGB);
UIImage *outImg = MatToUIImage(mat);
return outImg;
}
Swift
let imgMsg = message as! CompressedImageMessage // a build on the standard message, with format and data added
let backImg = OpenCVWrapper.decodeToImage(toImage: imgMsg.data!)
On Thu, Aug 29, 2019 at 12:12 AM unl0ck [email protected] wrote:
In my current robot project I worked out how to receive and load an array of Ints (rosbridge base64 encodes arrays, like you pointed out) so I can try to apply it to the image data.
My question was more about how do you suggest building a node that outputs a compressed image stream? Are you running a webcam + ros node combo that I might be able to replicate?
I used an raspi cam. but I hadn't time to implement it, sorry. my usecase was the raspi cam stream to my ipad, nearly real-time. I hoped I can use this way, because every other way whcih I found was to slow..
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW7QVMHNV72T435GFRLQG5EEFA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5NFKMA#issuecomment-526013744, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THW6IAR6A4EMV3VE7MCLQG5EEFANCNFSM4GQ5SZQQ .
This is all pretty interesting. I dug up a webcam so I might try out some things using the camera and the data provided in the first comment.
I'm most interesting in getting a compressed image on the ipad and publishing a small image (of floats)
Testing this afternoon - code now compiles...
swift
rbsImageSubscriber = rbsManager?.addSubscriber(topic:
"/subCompressed", messageClass: CompressedImageMessage.self, response: { (message) -> (Void) in
print("/subCompressed received")
*let* imgMsg = message *as*! CompressedImageMessage
print("format is \(imgMsg.format ?? "what?")") *//
Apparently we don't need/care about png vs. jpeg...*
*let* img = OpenCVWrapper.decode(toImage: imgMsg.data!)
*self*.imgDisplay.image = img
})
c++
- (UIImage ) decodeToImage:(uint8_t) buff {
*//+ (UIImage *) decodeToImage:(unsigned char ) buff {
cv::Mat mat = cv::imdecode( (cv::InputArray) buff, -1);
*if* ( mat.empty() ) {
*return* *nil*;
}
cv::cvtColor(mat, mat, cv::COLOR_BGR2RGB);
UIImage *outImg = MatToUIImage(mat);
*return* outImg;
}
On Thu, Aug 29, 2019 at 9:01 AM wesgood [email protected] wrote:
This is all pretty interesting. I dug up a webcam so I might try out some things using the camera and the data provided in the first comment.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wesgood/RBSManager/issues/9?email_source=notifications&email_token=AA3THW2PNMQYD3SIPOINBZDQG7CCVA5CNFSM4GQ5SZQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5OMQ4Q#issuecomment-526174322, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3THW7OD7YWYGVZU7WRJ4TQG7CCVANCNFSM4GQ5SZQQ .