xDrip-Experimental
xDrip-Experimental copied to clipboard
StephenBlackWasAlreadyTaken
Setup
To point your existing forks here you can
open your existing DexDrip forks and do:
git remote add experimental https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental.git
then
git fetch experimental
checkout a new branch if you are currently doing work on master (Please dont push up your master!!!)
git checkout -b DexcomShareSupport << (obviously put in what you are actually working on)
git push experimental DexcomShareSupport << (again, dont push to master, push to experimental yourbranch)
Just created the first pull request. reviews are welcome.
I have also noticed that bhandfast https://github.com/bhandfast has created a branch called: Minor-Fixes-and-Improvements.
I did not see a pull request so I did not comment on this, but here are my comments:
- Things looks good generally speaking. I hope the tables are more readable now, although I did not check that.
- as for:
public class TransmitterDataDuplicateDetector { public boolean isConsideredDuplicate(TransmitterData lastData, TransmitterData newData) { if(lastData == null) return false;
return lastData.raw_data == newData.raw_data &&
Math.abs(lastData.timestamp - newData.timestamp ) < (10000); } }
so: we already have a similar functionality: public static boolean almostEquals( TransmitterRawData e1, TransmitterRawData e2) { if (e1 == null || e2==null) { return false; } // relative time is in ms if ((Math.abs(e1.CaptureDateTime - e2.CaptureDateTime) < 120 * 1000 ) && (e1.TransmissionId == e2.TransmissionId)) { return true; } return false; }
In any case I must say that 10 seconds is definitely not enough. I have noticed that when there are different android devices they might be 30 seconds apart. Since we have a 5 minutes granularity, I use 2 minutes to be sure. Also the TransmissionId is the field that it's only meaning is to tell if it is the same packet. If one wants to check other fields, that is obviously ok from my side, but let's also check the transmission id.
Thanks Tzachi
On Tue, Feb 17, 2015 at 5:13 PM, Stephen Black [email protected] wrote:
To point your existing forks here you can open your existing DexDrip forks and do: git remote add experimental https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental.git then git fetch experimental checkout a new branch if you are currently doing work on master (Please dont push up your master!!!) git checkout -b DexcomShareSupport << (obviously put in what you are actually working on) git push experimental DexcomShareSupport << (again, dont push to master, push to experimental yourbranch)
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2 .
Ok, I take a look at it. Currently that code is only ported from the master. The code is essentially backported from another branch I have which both receives data from the BLE-kit and some raspberry PI devices with Wixels that broadcast raw data to the local LAN.
I'll test things out a bit more and create a PR when I think it's working ok.
How do this raspberry PI work? Is this some open code project? Do they have a way to broadcast to longer ranges than local lan?
thanks Tzachi
On Wed, Feb 18, 2015 at 12:30 AM, bhandfast [email protected] wrote:
Ok, I take a look at it. Currently that code is only ported from the master. The code is essentially backported from another branch I have which both receives data from the BLE-kit and some raspberry PI devices with Wixels that broadcast raw data to the local LAN.
I'll test things out a bit more and create a PR when I think it's working ok.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-74769103 .
Well, the only thing they do is to listen to the serial port from the Wixel (using the same software as @StephenBlackWasAlreadyTaken created). Then broadcast it the the local LAN in clear text. So I cannot really recommend this since this opens up for anyone to send data over the local LAN (at home, work, cafes etc) to DexDrip.
This need to be secured before making it public :)
Well, I guess that if this is python code, they can change that to tcp/ip in just a few more lines of code...
On Wed, Feb 18, 2015 at 12:48 AM, bhandfast [email protected] wrote:
Oh, and the tiny software on the PIs are a just a few lines of python.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-74772093 .
By the way, if the PIs can run java, than there is existing code that does the tcp/ip part, and also there is no need to change anything in dexdrip, it can just connect to them.
Please let me know if you are interested.
Thanks Tzachi
On Wed, Feb 18, 2015 at 12:56 AM, Tzachi Dar [email protected] wrote:
Well, I guess that if this is python code, they can change that to tcp/ip in just a few more lines of code...
On Wed, Feb 18, 2015 at 12:48 AM, bhandfast [email protected] wrote:
Oh, and the tiny software on the PIs are a just a few lines of python.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-74772093 .
Synchronize access to the DexCollectionService (though the service will probably never run concurrently in different threads).
On android, services are running on the UI thread. so, I don't see how there can be another thread running there. We should probably make sure that there are no other threads running, or if there are threads, we should understand where they are coming from and make sure that we are ok with it. synchronizing one class will not realy help.
So, if we are afraid from threading issues, we should add prints for that (maybe even asserts, although I'm afraid to add asserts to this "mission critical" code).
All that said, there is one thread that I have introduced, and I intend to make sure that it will not call any other part of the code soon.
Thanks Tzachi
On Wed, Feb 18, 2015 at 5:19 PM, Tzachi Dar [email protected] wrote:
By the way, if the PIs can run java, than there is existing code that does the tcp/ip part, and also there is no need to change anything in dexdrip, it can just connect to them.
Please let me know if you are interested.
Thanks Tzachi
On Wed, Feb 18, 2015 at 12:56 AM, Tzachi Dar [email protected] wrote:
Well, I guess that if this is python code, they can change that to tcp/ip in just a few more lines of code...
On Wed, Feb 18, 2015 at 12:48 AM, bhandfast [email protected] wrote:
Oh, and the tiny software on the PIs are a just a few lines of python.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-74772093 .
Yes, but I believe the BLE callbacks are from another thread. The documentation indicates this and the ADB log have different thread id on the callbacks.
Anyway, I agree that we should avoid synchronized blocks if we can. We could deliver the callbacks to the UI thread but what about those randomDelay calls? What's their purpose? We don't want to block the UI thread.
Soooo the random delay calls are a very hacky way to get around duplicate data (if for some reason you got in a state with two gatt servers spun up for the same device and both get the same callback for the characteristic getting updated.
So say we have transmission a and b but they are the same transmission, just from different gatt servers
We can check to see if a is not already in the database and on the other thread check to see that b is not in the database. Neither will be there at this point so it will try to write both of them! And since sqlite is not async these commands queue up behind each other which pretty much guarantees they all get in which we don't want.
So, I had two thoughts about reducing the race conditions of it all, either wait a few seconds and just grab the last entry in that table and process that which still seemed a little shaky or purposely delay one more than the other on their way in to shuffle the queries for existing records (presumably one would have completed before the other one checks to see if it already exists)
Im sure there is a way to do this all without being so hacky, I don't know Java though so I am using anything I can. Definitely open to suggestions. Also I think we definitely need to pull calibration math off the main thread, that stuff can lag. On Feb 19, 2015 4:51 AM, "bhandfast" [email protected] wrote:
Yes, but I believe the BLE callbacks are from another thread. The documentation indicates this and the ADB log have different thread id on the callbacks.
Anyway, I agree that we should avoid synchronized blocks if we can. We could deliver the callbacks to the UI thread but what about those randomDelay calls? What's their purpose? We don't want to block the UI thread.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75023648 .
Hi,
I have a similar problem in WixelReader.java. Since I'm ready to accept data from many sources, there is a chance that two receivers will get the same data.
My algorithm is very simple. When going up first packet goes in, but I remember it. Next packets goes in, if it is not almost equal to the last one that goes up.
Actually, since my receivers might delay packets for a long time (they only answer when they are asked), I also make sure that the time of the new packet is newer than the time of the old packet. Also, since there are a few clocks that are involved there, I check that the packets are not from the too far future. In other words, if due to clock differences there is a time gap of up to 2 minutes, I'm ready to accept the packet). if the time differance is bigger, than I assume it is a mistake and the packet does not go up.
you can see the code at WixelReader.run()
If we want to make it even more bullet proof, when going up, I should look what was the last packet in, and not allow any packet that matches it.
As for the threading issues. If we move the BLE callbaks to the main thread, than things will be synced from the fact that only the main thread is running, which means there are no races. If needed I can look on the ways of moving work to the main thread.
Hope this helps Tzachi
On Thu, Feb 19, 2015 at 3:55 PM, Stephen Black [email protected] wrote:
Soooo the random delay calls are a very hacky way to get around duplicate data (if for some reason you got in a state with two gatt servers spun up for the same device and both get the same callback for the characteristic getting updated.
So say we have transmission a and b but they are the same transmission, just from different gatt servers
We can check to see if a is not already in the database and on the other thread check to see that b is not in the database. Neither will be there at this point so it will try to write both of them! And since sqlite is not async these commands queue up behind each other which pretty much guarantees they all get in which we don't want.
So, I had two thoughts about reducing the race conditions of it all, either wait a few seconds and just grab the last entry in that table and process that which still seemed a little shaky or purposely delay one more than the other on their way in to shuffle the queries for existing records (presumably one would have completed before the other one checks to see if it already exists)
Im sure there is a way to do this all without being so hacky, I don't know Java though so I am using anything I can. Definitely open to suggestions. Also I think we definitely need to pull calibration math off the main thread, that stuff can lag. On Feb 19, 2015 4:51 AM, "bhandfast" [email protected] wrote:
Yes, but I believe the BLE callbacks are from another thread. The documentation indicates this and the ADB log have different thread id on the callbacks.
Anyway, I agree that we should avoid synchronized blocks if we can. We could deliver the callbacks to the UI thread but what about those randomDelay calls? What's their purpose? We don't want to block the UI thread.
— Reply to this email directly or view it on GitHub < https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75023648
.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75054974 .
I made a slightly different hack. I simply reject any duplicate packets within a time period. The first one I work with, then ignore any following ones for a minute. Since it should be at least 5 minutes between packets, this works well, even for multiple sources transmitting at the same time. I thought the random delays were contributing to the "jitter" in the graphs. It still exists now with dexbridge, because there is no determination time in the BLE connection happening, and I store the Dexcom packet for up to a minute to wait for the connection. Cheers
On Fri, Feb 20, 2015 at 1:34 AM, tzachi-dar [email protected] wrote:
Hi,
I have a similar problem in WixelReader.java. Since I'm ready to accept data from many sources, there is a chance that two receivers will get the same data.
My algorithm is very simple. When going up first packet goes in, but I remember it. Next packets goes in, if it is not almost equal to the last one that goes up.
Actually, since my receivers might delay packets for a long time (they only answer when they are asked), I also make sure that the time of the new packet is newer than the time of the old packet. Also, since there are a few clocks that are involved there, I check that the packets are not from the too far future. In other words, if due to clock differences there is a time gap of up to 2 minutes, I'm ready to accept the packet). if the time differance is bigger, than I assume it is a mistake and the packet does not go up.
you can see the code at WixelReader.run()
If we want to make it even more bullet proof, when going up, I should look what was the last packet in, and not allow any packet that matches it.
As for the threading issues. If we move the BLE callbaks to the main thread, than things will be synced from the fact that only the main thread is running, which means there are no races. If needed I can look on the ways of moving work to the main thread.
Hope this helps Tzachi
On Thu, Feb 19, 2015 at 3:55 PM, Stephen Black [email protected] wrote:
Soooo the random delay calls are a very hacky way to get around duplicate data (if for some reason you got in a state with two gatt servers spun up for the same device and both get the same callback for the characteristic getting updated.
So say we have transmission a and b but they are the same transmission, just from different gatt servers
We can check to see if a is not already in the database and on the other thread check to see that b is not in the database. Neither will be there at this point so it will try to write both of them! And since sqlite is not async these commands queue up behind each other which pretty much guarantees they all get in which we don't want.
So, I had two thoughts about reducing the race conditions of it all, either wait a few seconds and just grab the last entry in that table and process that which still seemed a little shaky or purposely delay one more than the other on their way in to shuffle the queries for existing records (presumably one would have completed before the other one checks to see if it already exists)
Im sure there is a way to do this all without being so hacky, I don't know Java though so I am using anything I can. Definitely open to suggestions. Also I think we definitely need to pull calibration math off the main thread, that stuff can lag. On Feb 19, 2015 4:51 AM, "bhandfast" [email protected] wrote:
Yes, but I believe the BLE callbacks are from another thread. The documentation indicates this and the ADB log have different thread id on the callbacks.
Anyway, I agree that we should avoid synchronized blocks if we can. We could deliver the callbacks to the UI thread but what about those randomDelay calls? What's their purpose? We don't want to block the UI thread.
— Reply to this email directly or view it on GitHub <
https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75023648
.
— Reply to this email directly or view it on GitHub < https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75054974
.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75061150 .
John Stevens "You are how you live, not what you have."
That unfortunately wouldn't prevent the issue of two gatt servers responding to the same exact same characteristic update, they would be too close together to rely on the db to help and are on different threads, maybe if we lock down a single observable with rxjava we could make them push back on to the same thread, I think that may be the best bet. The jitter in the graph is unfortunately the library I used bucketing values, if we cut it down to by second instead of my MS it may clear up some.
I also implemented queuing of the GATT, to avoid any issues of packets showing up at the same time. So, I can never get two or more exactly hitting at the same time. FIFO helps a lot with that. Cheers
On Fri, Feb 20, 2015 at 2:31 PM, Stephen Black [email protected] wrote:
That unfortunately wouldn't prevent the issue of two gatt servers responding to the same exact same characteristic update, they would be too close together to rely on the db to help and are on different threads, maybe if we lock down a single observable with rxjava we could make them push back on to the same thread, I think that may be the best bet. The jitter in the graph is unfortunately the library I used bucketing values, if we cut it down to by second instead of my MS it may clear up some.
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/DexDrip-Experimental/issues/2#issuecomment-75184564 .
John Stevens "You are how you live, not what you have."
Anyone know how to gap sync the xdrip? His phone was out of range and I want to get yesterday's in so on nightscout. Thank you.
tzachi and bhandfast both have implementations that allow sending the raw data over a private network and the potential of sending it over a public network was discussed above. It was suggested here that the data needed to be secured before making it public.
I am working on something similar and after reading this comment I was considering using AES but then I decided that maybe generating an SHA-2 hash on the raw data (appended with a secret key) might do the trick. It may also be applicable in some way to dexterity and other similar solutions.
There are a few SHA256 implementations in C but before I try to implement this on wixel-xdrip I was just wondering if there are any comments regarding this approach? For the guys above, the xDrip app would probably need to validate the hash. However, in my implementation I would be sending the data to a socket on a java PC uploader over the internet, which would then validate the hash before uploading it into a mongo db for use by the xDrip (using the pre-existing wifi network receiver functionality) and so no modification to xDrip app should be required. (Before you ask, my preference would be to upload to mongo directly but I don't think I can do that with my hardware for various technical reasons.)
Would there be any reservations about doing it this way or even am I over-engineering it? Also, do you think it will fit on the wixel (as I have a load of other stuff going on there as well)? I suppose there is one way to find out... lol
Sorry if this hijacks a semi-dormant thread but I have been mulling over the concerns regarding sending/receiving data in the clear since I saw it raised above. Also, I know I am focussing more on the sending technology but it may impact the xdrip-experimental branch if you decided that this is the way to go for securing received data over public networks from dexterity and other related projects.
Cheers
You should be aware that the wixel has build in encoding engine to simplify this. Personally, I do not feel there is any need to hash it from the wixel to app via BLE. But then again, I use xBridge which uses a binary protocol, rather than the plain text format of the wixel-xdrip. I wouldn't use the wixel-xdrip, as it lacks many of the features I wanted, and included in xBridge.
I would think it much more important to ensure the data from xDrip to NS is encrypted, rather than from the bridge to app. Especially as the Dexcom transmitter sends it unencrypted. Cheers
On Wed, Sep 23, 2015 at 1:42 AM, Keith Garland [email protected] wrote:
tzachi and bhandfast both have implementations that allow sending the raw data over a private network and the potential of sending it over a public network was discussed above. It was suggested here that the data needed to be secured before making it public.
I am working on something similar and after reading this comment I was considering using AES but then I decided that maybe generating an SHA-2 hash on the raw data (appended with a secret key) might do the trick. It may also be applicable in some way to dexterity and other similar solutions.
There are a few SHA256 implementations in C but before I try to implement this on wixel-xdrip I was just wondering if there are any comments regarding this approach? For the guys above, the xDrip app would probably need to validate the hash. However, in my implementation I would be sending the data to a socket on a java PC uploader over the internet, which would then validate the hash before uploading it into a mongo db for use by the xDrip (using the pre-existing wifi network receiver functionality) and so no modification to xDrip app should be required. (Before you ask, my preference would be to upload to mongo directly but I don't think I can do that with my hardware for various technical reasons.)
Would there be any reservations about doing it this way or even am I over-engineering it? Also, do you think it will fit on the wixel (as I have a load of other stuff going on there as well)? I suppose there is one way to find out... lol
Sorry if this hijacks a semi-dormant thread but I have been mulling over the concerns regarding sending/receiving data in the clear since I saw it raised above. Also, I know I am focussing more on the sending technology but it may impact the xdrip-experimental branch if you decided that this is the way to go for securing received data over public networks from dexterity and other related projects.
Cheers
— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/issues/2#issuecomment-142328036 .
John Stevens "You are how you live, not what you have."
Thanks John. Yes, hashing over BLE does seem like overkill but I am actually talking about securing the data being sent from a remote monitor over the internet using TCP/IP. The point that bhandfast was making was that someone else anywhere on the internet could go to the same IP address and inject random fake packets if they so wished.
I'm not sure exactly what the binary protocol on the xbridge is that you refer to but I think you mean that it is just sending the raw bytes which are not particularly human-readable. However, it would seem fairly trivial to spoof these over a public network if someone wanted to. I was suggesting that if I was adopt an approach using AES or SHA that it would be something that might be also usable on the xbridge to secure remote connections over the internet.
It is a bit of extra work to implement this functionality but I can see the argument for it. I would be interested in knowing more about the built-in support on the wixel for hashing / encryption as I would just like to generate a checksum to send along with my data. If the wixel has built-in hashing functions then that would be great.
I'm not sure if I have this right but the I think the wixel-xdrip is the wixel/BLE combo and the xbridge is the wixel/USB combination? My "xdrip2g" is actually a wixel/fona combination that sends data directly from the wixel over a GPRS connection to a TCP socket on a remote computer. It is that connection that it seems that I would need to be "secured before making it public". Personally I am happy to implement "security through obscurity" for the moment but I can see how people might want it to be a bit more robust before releasing it into the wild.
As the fona does not support SSL connections, it is a bit of a nightmare trying to update mongolab databases directly. However, that would be a direct secure connection if I could manage that and I wouldn't have to worry about any of this hashing stuff. For the moment I am going with a plaintext TCP/IP socket connection to a PC (using a dynamic dns and port forwarding) to get the data off the wixel. After that, everything is secure as it will upload to the mongo database using the usual secure methods. So it is just the first hop from the wixel to the PC which is "vulnerable".
Have you any more info on the "secure encoding engine" please? I had a browse in the wixel-sdk and did a bit of a google but couldn't find any reference to it. I'm just looking for an easy way to generate a hashed checksum to verify that the data is not being spoofed. Some sample code would be nice! ;)
Thanks very much for the swift reply. It is great to get some feedback and the info you have provided is really useful.