Subscribe and Notify are not working on Linux

[dsteinman]

Hi I really appreciate your work to get noble working again on the Raspberry Pi. The code seems to compile alright, and my BLE device (an ESP32) can connect, but none of the methods of the characteristic objects seem to work.

characteristic.on('data', function(data) {
   // no events
});
characteristic.subscribe(function(err) {
   // never gets called
});
otherCharacteristic.read(function(error, data) {
   // never gets called 
});
otherCharacteristic.write(buffer, false, function(error) {
   // never gets called and nothing received on device
});

My code works on MacOSX using the "noble-mac" fork but is not working with yours on a Pi3 (respbian), so perhaps there is something still missing for this to work on the Pi3 ?

[rzr]

Please can you share logs of examples in:

https://github.com/abandonware/noble/tree/master/examples

[dsteinman]

The peripheral-explorer.js was the most helpful for my situation and here is the result when connecting my device:

(node:19638) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
peripheral with ID 240ac4121576 found
  Local Name        = ESP32
  TX Power Level    = -21
  Service Data      = []
  Service UUIDs     = c799a4c153ae45b6a37ae277e7485b5a

services and characteristics:
1801 (Generic Attribute)
  2a05 (Service Changed)
    properties  indicate
1800 (Generic Access)
  2a00 (Device Name)
    properties  read
    value       4553503332 | 'ESP32'
  2a01 (Appearance)
    properties  read
    value       0000 | ''
  2aa6 (Central Address Resolution)
    properties  read
    value       00 | ''
c799a4c153ae45b6a37ae277e7485b5a
found notify
  234e7d9d2cef4026891f9e893edfb06e
    properties  notify
  848f7ba939f8408faa85216c35362d79
    properties  write
  64289ec7619042c6a172beba419e6752
    properties  read, write
    value       4d59455350 | 'MYESP'

This all looks good to me. It finds my 3 characteristics (1 notify, 1 write, and 1 read/write). It turns out I was mistaken, the read() method is working correctly because that last line result "MYESP" is coming from my BLE firmware. The write() also works.

But then I tried modifying that example by adding a .subscribe() and .on('data') and then it fails, I replaced lines 131 through 133 with the following to add a subscribe() and 'data' event listener:

} else if (characteristic.properties.indexOf('notify') !== -1) {
	console.log('found notify');
	
	characteristic.on('data', function (data) {
		console.log('data:::', data); // never gets called
	});
	
	characteristic.subscribe(function () {
		console.log('subscribed'); // never gets called
	});
	
	console.log('after subscribe'); // does get called but all BLE communication stops here
	
	callback();
} else {
	callback();
}

The example continues to run, but all communication to/from the BLE device seems to halt after that .subscribe() call is made.

I also realized this noble module is indeed working on my Mac as well, so I was able to go back and forth between my Mac and a Raspberry Pi 3 to determine that this is exactly where the problem lies and it works on my Mac, but fails on my Raspberry Pi3. The .subscribe() call never returns, and somehow is shutting down.

I'll probably drill down in a bit to that that .subscribe() call and find the exact root cause.

[dsteinman]

I've been able to locate the probable cause.

https://github.com/noble/noble/blob/master/lib/hci-socket/gatt.js

Gatt.prototype._queueCommand = function(buffer, callback, writeCallback) {
  this._commandQueue.push({
    buffer: buffer,
    callback: callback,
    writeCallback: writeCallback
  });

  if (this._currentCommand === null) {
    while (this._commandQueue.length) {
      this._currentCommand = this._commandQueue.shift();

      this.writeAtt(this._currentCommand.buffer);

      if (this._currentCommand.callback) {
        break;
      } else if (this._currentCommand.writeCallback) {
        this._currentCommand.writeCallback();

        this._currentCommand = null;
      }
    }
  }
};

this._commandQueue[] keeps filling up and not getting cleared out properly because of the way the .callback property is handled.

if (this._currentCommand.callback) {
        break;

That line stops the command queue, and then waits for a subsequent message to be read some time later in onAclStreamData() before proceeding.

Gatt.prototype.onAclStreamData = function(cid, data) {
    ...
    this._currentCommand.callback(data);  // line 132
    
    this._currentCommand = null;

There seems to be several ways that this can fail, including simply if no data is received from the BLE device which appears to be the case when a .subscribe() is called. It's waiting for incoming data and if none is received then all subsequent commands continue to be queued. At least that's how I'm understanding it.

I might need some help to rearrange this code in such a way that this can be avoided. I have simplified my Arduino firmware sketch down to the basics and I can reproduce the same issue consistently working on the Mac version, and failing on a Raspberry Pi. I presume it will also fail on any Linux/Windows/FreeBSD system because the 'bluetooth-hci-socket' library is used for all of them, where as the Mac uses a native binding.

My sketch and the modified noble example peripheral-explorer.js here:

https://github.com/dsteinman/noble-ble-firmware-esp32

Basically it sets up a counter and notifies each second, but the .subscribe() never returns. I'm not sure if any data is supposed to be received from a BLE device when you subscribe to a notification characteristic.

[rzr]

If you can reproduce this on any upstream version, please file a bug there too and crosslink it here Thx

[dsteinman]

I would love to do so but unfortunately BluetoothHciSocket doesn't compile for me on the main codebase at https://github.com/noble/noble

binding.target.mk:99: recipe for target 'Release/obj.target/binding/src/BluetoothHciSocket.o' failed
make: *** [Release/obj.target/binding/src/BluetoothHciSocket.o] Error 1
make: Leaving directory '/home/pi/nobletest/noble/node_modules/bluetooth-hci-socket/build'

Perhaps the maintainers of /noble/noble need to be contacted to merge the abandonware changes in or convince them to tombstone that repo and use this one instead.

[rzr]

What about earlier node version?

Relate-to: https://github.com/noble/noble/pull/851

[dsteinman]

I'm on v10.15.3 on my Pi at the moment, and pretty sure it failed with v8 as well.

There's already bugs posted on the main repo, and I just posted this new one:

https://github.com/noble/noble/issues/886

Has there been any response from any maintainer of that repo?

[marksyzm]

I have the same issue on macOS :(

[dsteinman]

I should have updated this.

I was able to avoid this problem by migrating my Arduino firmware to use Adafruit's Bluefruit BLE library rather than the "old" BLEPeripheral library which is no longer maintained. Bluefruit is maintained and does work as expected with noble (this abandonware fork).

[jordanorc]

I'm facing the same problem here, but I have no choice to migrate device firmware, once it is proprietary. Does anyone have a workaround for this?

[dsteinman]

@LukasBombach is taking on the task of rewriting noble and his port is probably worth a try.

Discussion here: https://github.com/noble/noble/issues/874#issuecomment-526241295

Repo here: https://github.com/LukasBombach/sblendid/

[jordanorc]

Hi @dsteinman, thank's for the reply. I'll try it!

[cchaloin]

Same thing. I'm getting an error with noble 1.9.2-5 and Node 12.10 running on Rasbian Buster (Rasp Pi 4)

Nor characteristic.subscribe/read neither characteristic.notify/data are working.

But interestingly, without doing any subscribe/Notify(true) , characteristic.on('read', function(data, isNotification)) is triggered with isNotification set to 'true' when that NOTIFY only characteristic is getting new datas. Not sure it is expected to work this way.

Error returned when trying to subscribe:

internal/buffer.js:72
  throw new ERR_OUT_OF_RANGE(type || 'offset',
  ^

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 2. Received 4
    at boundsError (internal/buffer.js:72:9)
    at Buffer.readUInt16LE (internal/buffer.js:229:5)
    at Gatt.<anonymous> (/root/youdome/node/node_modules/@abandonware/noble/lib/hci-socket/gatt.js:625:24)
    at Gatt.onAclStreamData (/root/youdome/node/node_modules/@abandonware/noble/lib/hci-socket/gatt.js:132:26)
    at AclStream.emit (events.js:214:15)
    at AclStream.push (/root/youdome/node/node_modules/@abandonware/noble/lib/hci-socket/acl-stream.js:35:10)
    at NobleBindings.onAclDataPkt (/root/youdome/node/node_modules/@abandonware/noble/lib/hci-socket/bindings.js:286:15)
    at Hci.emit (events.js:209:13)
    at Hci.onSocketData (/root/youdome/node/node_modules/@abandonware/noble/lib/hci-socket/hci.js:497:14)
    at BluetoothHciSocket.emit (events.js:209:13)

[arthare]

This may not be the solution for other people, but it was for me, and this thread was the strongest match to my symptoms.

The wrapper library around my Noble instance tended to discover characteristics frequently.

Extra round of characteristic discovering meant that the Noble._characteristics array/map would get new Characteristic instantiations for a given peripheral/service/characteristic combo. Obviously the new instantiations would not have my .on('data') listeners that I previously attached still attached to them, and so my .on('data') listeners wouldn't get hit when new notifications appeared.

This meant that when callbacks came back from the native code, noble.js would look up the most-recent version of the Characteristic objects representing that peripheral/service/characteristic combo, do a .emit('data') on that object, but that .emit() would go nowhere, since my code would have never seen those particular characteristic objects.

I put in a quick hack (which might be a quick solution for me, if not for everyone) in noble.js's onCharacteristicsDiscover function that looks like this:

Noble.prototype.onCharacteristicsDiscover = function (peripheralUuid, serviceUuid, characteristics) {
  const service = this._services[peripheralUuid][serviceUuid];

  if (service) {
    const characteristics_ = [];

    for (let i = 0; i < characteristics.length; i++) {
      const characteristicUuid = characteristics[i].uuid;

      let characteristic;
      try {
        const alreadyHad = this._characteristics[peripheralUuid][serviceUuid][characteristicUuid];
        if(!alreadyHad) {
          throw alreadyHad;
        }
        characteristic = alreadyHad;
      } catch(e) {
        characteristic = new Characteristic(
          this,
          peripheralUuid,
          serviceUuid,
          characteristicUuid,
          characteristics[i].properties
        );
      }

      this._characteristics[peripheralUuid][serviceUuid][characteristicUuid] = characteristic;
      this._descriptors[peripheralUuid][serviceUuid][characteristicUuid] = {};

      characteristics_.push(characteristic);
    }

    service.characteristics = characteristics_;

    service.emit('characteristicsDiscover', characteristics_);
  } else {
    this.emit('warning', `unknown peripheral ${peripheralUuid}, ${serviceUuid} characteristics discover!`);
  }
};

And that halved the # of new Characteristic instantiations, as well as solved my problem with my notifications not arriving.

I'm not sure if this is the correct solution - it feels weird that discoverCharacteristics calls may invalidate all your existing characteristics. But maybe my code shouldn't discoverCharacteristics as frequently as it did.

[qcasey]

Same issue here, @arthare I tried your solution but still no luck. subscribe's callback is never invoked and data completely stops, as if the adapter is frozen or something.

Central is 5.6.7-1-MANJARO, peripheral is Arduino Nano 33 BLE

This is what I get from debug, not sure how to interpret:

Setting up notifications for 3c9d47c71caa4c1db40a962c9ca6c327
  att d0:a0:18:72:98:ba: write: 08230025000229 +18ms
  hci write acl data pkt - writing: 02010e0b000700040008230025000229 +19ms
  hci onSocketData: 02012e090005000400010823000a +41ms
  hci   event type = 2 +0ms
  hci           cid = 4 +0ms
  hci           handle = 3585 +0ms
  hci           data = 010823000a +0ms
  att d0:a0:18:72:98:ba: read: 010823000a +41ms
  att d0:a0:18:72:98:ba: write: 08260028000229 +1ms
  hci write acl data pkt - writing: 02010e0b000700040008260028000229 +1ms
  hci onSocketData: 02012e090005000400010826000a +29ms
  hci   event type = 2 +0ms
  hci           cid = 4 +0ms
  hci           handle = 3585 +0ms
  hci           data = 010826000a +0ms
  att d0:a0:18:72:98:ba: read: 010826000a +29ms

EDIT: I dug further into the gatt.js file and found my issue was more aligned with #887. I still think it should report a problem in the debug log when data[0] indicates an error

[buildog]

@qcasey did you found a fix for this issue ? @LukasBombach may have an idea to solve it until sblendid linux support release https://github.com/LukasBombach/sblendid/projects/3 ?

[qcasey]

No, but I'd still be very interested in a fix. Let me know if you had any ideas

[LukasBombach]

@buildog sorry, I couldn't find anyone having fixed this. If you want to do things yourself, you'd need to implement a noble adapter using Linux's Bluez API.

[DonkDH]

Hey, I am having similar issues on my Raspberry Pi Zero of the event for notify/subscribe/char.on('data;,..) are not firing. Guessing no movement on this? Thanks

[buildog]

No, best move is to abandon noble.

We moved to https://www.npmjs.com/package/node-ble

It works on RPi0

[DonkDH]

Cool thank you, I will give that a go over the weekend!

[arthare]

My team was too deep into noble before this stuff came up, so we've been gradually fixing noble. There's been a bunch of other little spots like my earlier suggestion, where it needlessly instantiates a new Peripheral, or Characteristic, or such, and so the emit('whatever')'s don't find their way to the code that's listening via a .on('whatever') to the old instance of the peripheral.

I've got it to the point where 8 separate processes can talk to the noble websocket server (ws-slave.js) and perform firmware updates to our devices ~400 times before the first failure. But I've probably made some changes that only really work for our team.

node-ble sounds tempting though.

[rabee05]

I am having the same issue between ESP32 and Ubuntu. The main issue with descriptor 0x2902. I was told that I need to send write value 0x01 or 0x02 to 0x2902 descriptor on characteristic that will send notify. I tried to send the values using noble

descriptors[0].writeValue(Buffer.from(['1'], 'hex'), (err) =>console.log(err));

Any idea how to send some values the descriptors?

Best, RZ

[rabee05]

I ended up changing the code inside the ESP32 because subscribe/Notify doesn't work at all. We do not always have access to the embedded code. I hope to have a fix soon. Please share with us any workaround available.

Best, Rabee

[WTX1]

Try this

notifyCharacteristic.once('descriptorsDiscover', async (descriptors) => {
    const buff = Buffer.from([0x00, 0x01]); 
    descriptors[0].writeValue(buff, (error) => {
        if (error) console.log(error);
    });
    await notifyCharacteristic.subscribeAsync();
});

notifyCharacteristic.discoverDescriptors((err) => {
    if (err) console.log(err)
});

You look for 0x2902 descriptor on your notification characteristic then send [0x00, 0x01]. After that you can subscribe to the notification and receive data.

It worked for me on RPi 4 (debian buster).

[mabuonomo]

I resolve using the following two envs NOBLE_MULTI_ROLE=1 and NOBLE_REPORT_ALL_HCI_EVENTS=1 (see the documentation https://github.com/abandonware/noble)

[DrCWO]

Hi @WTX1,

Try this

notifyCharacteristic.once('descriptorsDiscover', async (descriptors) => {
    const buff = Buffer.from([0x00, 0x01]); 
    descriptors[0].writeValue(buff, (error) => {
        if (error) console.log(error);
    });
    await notifyCharacteristic.subscribeAsync();
});

notifyCharacteristic.discoverDescriptors((err) => {
    if (err) console.log(err)
});

You look for 0x2902 descriptor on your notification characteristic then send [0x00, 0x01]. After that you can subscribe to the notification and receive data.

It worked for me on RPi 4 (debian buster).

I tried to do it as you proposed but I don't see a 0x2902 descriptor but only a 0x2908 descriptor :-(

The device I like to get data from is a Microsoft Surface Dial being a HID device. I disabled HID in /lib/systemd/system/bluetooth.service (--noplugin=sap,input,hog) to be able to see the HID service. This worked. I also can read for example the Report Map characteristic.

But I have no idea to get data from the Report characteristic and wonder where to find the Client Characteristic Configuration Descriptor 0x2902.

Any kind of support would be appreciated. Best DrCWO