svd-xbee
svd-xbee copied to clipboard
jankolkmeier
Fix for floating point numbers
Casting data to ascii wreaks havoc later on when converting to binary. I propose to keep it as Buffer for as long as possible.
I have written up a small pastebin explaining the problem: http://pastebin.com/cTa1xgQQ
Just in case pastebin file dissapears in a month or so, here's a copy: Original problem:
I am sending a float from arduino to node server via Xbee S2.
There is a strange behavior where the value node reports shows up normal from 0 to 16 on arduino shows 0 to 16 on nodejs from 16 to 32 on arduino shows 8 to 16 on nodejs from 32 to 64 on arduino shows as 32 to 64 on nodejs from 64 to 128 on arduino shows as 32 to 64 on nodejs
I have obtained binary output from both sides:
Names are shortened for sake of alignment: ard: Arduino nod: Node
ard 16.20: 10100100 10011001 10000001 10000001 nod 8.06: 00100100 00011001 00000001 10000001
ard 15.80: 01110100 11001100 11111100 10000001 nod 15.44: 01100100 01001100 01111100 10000001
The leftmost bit in each byte is off on receiving end.
Hey igor, thank you for contributing!
Just to be sure I understand you right: the bytes itself transmit just fine, it is just that the bytes after ascii conversion are messed up? If you parse the 'binary' bytes to float everything works just fine?
While your PR would fix the issue, then, It would probably break some backwards compatibility.
For now, I'd say just get the extra variable that is emitted on the data event, which is the original packet and contains the raw data bytes:
node.on('data', function(data, packet) {
// This should contain the original bytes:
packet.rawData;
});
Please tell me If this does fix the issue here, or if I'm understanding it wrong!
Cheers, Jan
Indeed, the bytes transmit just fine and rawData to float works correctly. Took good while to find the cause.
Somehow conversion to ascii string strips off one bit, diving the value by half in certain ranges. I find it interesting that bits are only stripped from some bytes and not the others (the last of the 4 bytes is unaffected), but this could be due to my small sample range and lack of excessive testing.
I have not found any adverse side effects with this PR yet, in fact, it has fixed few other issues. for example, data[0] now correctly returns integer (sending 1), where as before it was returning "" (non viewable character), making it impossible to use in switch statements.
After submitting the pull request, I have noticed your comment there, saying the packet should be passed as buffer all the way :)
I will do some more testing extracting various data types from packet data with PR change and report back. Do you know of any projects that would make good testing grounds?
rawData value would be acceptable, but I'm not sure it's the best solution. Buffers are much much easier to manipulate than ascii strings and they can be converted at any time if needed. Modules like jParser make handling it very convenient.
Looking at http://nodejs.org/api/buffer.html#buffer_buffer we see indeed that 'ascii' only uses 7 bits! It also says that 'binary' will eventually be deprecated, but still, it is probably the best option for now.
jParser sure is a nice option - but almost overkill for this task. If you look at https://github.com/jouz/xbee-api, you will see that I have rewritten the package parsers and builders using buffers from start to end (also allowing for streaming support).
It is tested and written cleaner than svd-xbee. It does, however, not provide the higher level functionalities of svd-xbee (managing nodes, response callbacks, abstraction of AT commands, etc.). My plan is still to rewrite svd-xbee using xbee-api as backend... but the time.... ;)
That explains it. I'm not clear how to insert binary data into Buffers after they deprecate 'binary' option yet. Will need to play around once they do it.
I did have a brief look at xbee-stream, but as you said it doesn't yet provide the amazing functionality of managing nodes and callbacks, so I settled for svd-xbee.
So far svd-xbee has been absolute joy to use (apart from the little float bug :) ). I am moving from python backend to nodejs and the switch has been much easier than expected.
What I liked about jParser is ability to define the all the structures in one file. I believe it also does reverse packaging as well for responses. https://github.com/bryanpaluch/home-monitor/blob/master/sensor-communicator/lib/sensorData.js Here's an example from Bryan Paluch's thermostat.
I'll have a deeper look at xbee-api.
After digging around the svd-xbee code little more, I have realised that I misunderstood your original statement about packet.rawData.
I did not know it was already implemented and it does work as correctly. This pull request can be closed.
Could a note on 7-bit ascii conversion be added somewhere?
Some note would be good, yes! But I'm still not sure if the problem is dealt with properly already.
For example, I just noticed that when building a package to send, I convert it to a Buffer also using ascii (see here). I'm not sure why I made that choice in the first place, but I wonder if information gets lost here, too... Have you made any observations like that (for example when sending floats to your arduino)?
I spent about 5 hours last night trying to send data back to Arduino with no luck.
I am only getting ApiId 0x8B on Arduino, which is the ack, and nothing else. The code on Arduino is taken from xbee example and exact same code works in my other project, so I don't think it's the arduino side.
Not yet 100% on what's happening there. Oscilloscope shows data flowing from tx pin of xbee, but it could the above mentioned ACKs (my Nodejs server is setup to send test data back as soon as it receives a packet).
console.log of frame byte array for node.send(data) looks correct, the structure conforms to digi xbee api spec.
If svd-xbee sends 0x10 frame type, should should this not be the same ApiId Arduino receives? Arduino is waiting for 0x90 frame (ZB_RX_RESPONSE type).
I'll upload my server.js, maybe i'm simply not using svd-xbee correctly.
Yes, on the TX line you would expect the 0x8B packet to be sent (from xbee on arduino to xbee on node). If you measure that, you'd expect the 0x90 packet to come out RX the same moment. Maybe you haven't been accurate with your description, but to me it seems you're switching RX and TX lines around on the arduino side.
Ok, more testing and some results. I'm pretty sure I have pinpointed the problem spot. It's address of the node.
Broadcasting works (there is a tiny bug in the broadcasting function, but that's not the point, will submit a separate PR). I can see the data on oscilloscope and most importantly Arduino sees it too and correctly processes it. It also seems to correctly extracts the float from the payload, so that's good. This rules out the arduino side of things for me. it also rules out the data part of the payload.
When using xbee.nodes[address].send(data) however, there is no data coming out of receiving xbee. The only thing different between the 2 packets is the remote64 and remote16 address values. Could it be that?
Got it!
/lib/xbee.api.js
lines 136 to 143.
while generating 'res', the second byte of the address (0x13) gets changed to (0x33). Address no longer matches destination xbee = no data received.
Looks like my coordinator xbee is setup in api mode 2. I have tried to run receiving xbee in same mode, and it made no difference.
Update:
I'm happy to report that everything is now working well. Finaly :)
Thanks for that other commit, well spotted!
So what made this work in the end? Not a bug, or just the one in the broadcast function? Maybe I can update the documentation to prevent such errors in the future?
The 'fix' was very simple:
https://github.com/jouz/svd-xbee/blob/master/lib/xbee-api.js#L144
line 144 changed from: return new Buffer(res, 'ascii'); to return new Buffer(packetdata, 'ascii');
This avoids the whole escaping block. The exact line that causes problems is 141: res.push(packetdata[p] ^ 0x20);
it changes byte 0x13 to 0x33. in the remote64 address.
Should the escaping only be done on the data part of the packet?
Having a look at documentation, this conversion is correct, but for some reason did not work for me with apiMode 2 on receiving xbee.
Would the correct solution for my case is use api mode 1 on both sides?
Update:
Oh and the 'ascii' type when creating a buffer seems to make no difference at all. I tried 'hex', 'binary' and null. all return same bytes in the buffer
Escaping only concerns the communication over the serial line! In your case, the "receiving" xbee has nothing to do with the api mode you choose in svd-xbee! That only concerns the xbee that you talk to directly through serial. So the "sending" xbee must have AP=2 when using escaping (this seems not to be the case now for you). The other xbees (connected to arduinos or whatever) can have whatever other AP setting that is convenient for you! For more information, read (especially the last part of) this doc: http://www.digi.com/support/kbase/kbaseresultdetl?id=2199
On Thu, Jan 9, 2014 at 12:39 PM, Igorshp [email protected] wrote:
The 'fix' was very simple:
https://github.com/jouz/svd-xbee/blob/master/lib/xbee-api.js#L144
line 144 changed from: return new Buffer(res, 'ascii'); to return new Buffer(packetdata, 'ascii');
This avoids the whole escaping block. The exact line that causes problems is 141: res.push(packetdata[p] ^ 0x20);
it changes byte 0x13 to 0x33. in the remote64 address.
Should the escaping only be done on the data part of the packet?
Having a look at documentation, this conversion is correct, but for some reason did not work for me with apiMode 2 on receiving xbee.
Would the correct solution for my case is use api mode 1 on both sides?
— Reply to this email directly or view it on GitHubhttps://github.com/jouz/svd-xbee/pull/16#issuecomment-31922958 .
Ah, didn't know the that. Thanks!
I tried to stay away from switching apimode on the coordinator as I thought it may conflict with my other modules in the house (they're tucked away and not as easy to get to).
Great, so it's a simple xbee config change on my part.
Should the library be getting api_mode from xbee configuration rather than hardcoded value?
Update: Sigh, and you even have a paragraph on main page saying this module works with API mode 2 :) Good old case of "When all else fails, read the god damn instructions" :)
We'd have to check whether the request and response for querying the ATAP parameter could contain any bytes that would need escaping (and make sure that for example the frameid doesn't equal a byte to escape).
If not, that would be possible, by appending another parameter to the other ones that I read in readParameters (see https://github.com/jouz/svd-xbee/blob/master/index.js#L236 ), and then setting self.options.api_mode based on response that is stored in parameters value just before "initialized" is emitted.
I'm just not sure if that would be considered good practice, as the other requests in readParameters also may require the AP mode to be set correctly in the first place. Maybe if we query AP first and set api_mode immediately?
On Thu, Jan 9, 2014 at 2:53 PM, Igorshp [email protected] wrote:
Ah, didn't know the that. Thanks!
I tried to stay away from switching apimode on the coordinator as I thought it may conflict with my other modules in the house (they're tucked away and not as easy to get to).
Great, so it's a simple xbee config change on my part.
Should the library be getting api_mode from xbee configuration rather than hardcoded value?
— Reply to this email directly or view it on GitHubhttps://github.com/jouz/svd-xbee/pull/16#issuecomment-31932619 .