[Drag event] Hard/impossible to access all edge pixels in some situations

[thyttan]

Affected hardware version

Bangle 2

Your firmware version

2v16.45

The bug

When using Dragboard or Touch Keyboard (with the one-to-one input setting enabled) I have noticed I will pretty often have a hard time accessing characters closest to the edge of the screen when draging.

On Dragboard for example I sometimes find it's hard to access the bottom red rectangle to select letters if draging from further up the screen. This doesn't happen if I haven't calibrated the screen, but if I have it can become impossible to drag back down to the red rectangle to select letters. (calibrated like this while testing now: touchX1: -2, touchY1: 9, touchX2: 195, touchY2: 177)

I think I remember my problem with Touch Keyboard was present even on uncalibrated screen. I'll try again and come back with an update.

I've written this script below for seeing how the drag event acts close to the edges.

// Script for documenting the first/last pixel reported as touched on a drag from outside a screen edge, in on the screen and exiting the screen via the same edge.
//Upload to RAM in Web IDE and see the output in the console window

var lastB = 0;
var enterX, enterY, exitX, exitY, deltaX, deltaY;

function dragHandler(e) {

  // EDIT: Added lines to get direct visual feedback.
  g.clear().setColor(1,e.b,e.b).drawLine(0,e.y,g.getWidth(),e.y).drawLine(e.x,0,e.x,g.getHeight());

  if (lastB==0){
    print('enter via edge:');
    print(e);
    enterX = e.x;
    enterY = e.y;
    // g.dump();  // Uncomment to get graphics dumps to console field automatically.
  }

  if (e.b==0) {
    print('exit via same edge:'); 
    print(e);
    exitX = e.x;
    exitY = e.y;
    
    deltaX = exitX - enterX;
    deltaY = exitY - enterY;
    print('Delta x:', deltaX);
    print('Delta y:', deltaY);
    // g.dump();  // Uncomment to get graphics dumps to console field automatically.
    print('\n');
  }
  
  lastB=e.b;
}

Bangle.on('drag', dragHandler);

My findings are roughly (deltas given without sign):

• Top edge seems to not have too much problem, delta close to zero.
• Bottom edge I get a delta between 10-17 most of the time
• Left edge I get a delta between 6-10 most of the time
• Right edge I get a delta between 6-15 most of the time

I did the measurements with the touch screen uncalibrated.

I don't know exactly what solution I'm after here. But:

• I'd like to make sure Dragboard and Touch Keyboard works better. Maybe I can achieve this in Dragboard by moving the touch boundary between the red and green rectangles.
• Ideally something could be done so that deltaX and deltaY respectively are close to zero when entering and exiting an edge, when running the code above. But I suspect this is outside of espruino so could be hard.

Here is an output where I did top edge, bottom edge, left edge and right edge in that order (EDIT: I redid these with uncalibrated screen):

enter via edge:
{ "x": 102, "y": 9, "b": 1, "dx": 0,
  "dy": 0 }
exit via same edge:
{ "x": 101, "y": 12, "b": 0, "dx": 0,
  "dy": 0 }
Delta x: -1
Delta y: 3

enter via edge:
{ "x": 99, "y": 187, "b": 1, "dx": 0,
  "dy": 0 }
exit via same edge:
{ "x": 92, "y": 177, "b": 0, "dx": 0,
  "dy": 0 }
Delta x: -7
Delta y: -10

enter via edge:
{ "x": 11, "y": 93, "b": 1, "dx": 0,
  "dy": 0 }
exit via same edge:
{ "x": 9, "y": 70, "b": 0, "dx": 0,
  "dy": 0 }
Delta x: -2
Delta y: -23

enter via edge:
{ "x": 191, "y": 114, "b": 1, "dx": 0,
  "dy": 0 }
exit via same edge:
{ "x": 179, "y": 112, "b": 0, "dx": 0,
  "dy": 0 }
Delta x: -12
Delta y: -2

Installed apps

No response

[thyttan]

Updated the script above to also give visual feedback (vertical/horizontal lines) on the watch when testing.

[thyttan]

Here is what it looks like for me when the screen is uncalibrated touchX1: 0, touchY1: 0, touchX2: 160, touchY2: 160 (entering and exiting edges in order top, bottom, left, right):

Show prntscreens

[thyttan]

Here is what it looks like for me when the screen is calibrated touchX1: -3, touchY1: 19, touchX2: 192, touchY2: 176 (entering and exiting edges in order top, bottom, left, right):

Show prntscreens

The last one here where I try to access the right part of the screen is the most notable one.

[gfwilliams]

For enter/exit where the value is ~15px from the edge of the screen when calibrated - I'd say that's totally acceptable and you just need to cope with that in code.

If we calibrate the screen so it matches with your finger position then probably the 'edge' isn't going to be exactly on 0px, and I don't see there's a way around that.

But for the right-hand side, what happens if you just draw the coordinates?

Bangle.on('drag', e => {
  g.setPixel(e.x,e.y);
});

This is what I get when going off the edge of the screen:

I tried your code here and it works fine for me, I see stuff like:

enter via edge:
{ "x": 161, "y": 65, "b": 1, "dx": 0,
  "dy": 0 }
exit via same edge:
{ "x": 158, "y": 84, "b": 0, "dx": 0,
  "dy": 0 }
Delta x: -3
Delta y: 19

So maybe you could just try recalibrating? It's entirely likely you just didn't press right on the coordinates, and so the screen is now slightly uncalibrated so you can't reach the RHS

[thyttan]

For enter/exit where the value is ~15px from the edge of the screen when calibrated - I'd say that's totally acceptable and you just need to cope with that in code.

I can buy that. But it's weird the watch can register closer to the edge when entering and not be able to access the same pixels when draging back off the same edge?

But for the right-hand side, what happens if you just draw the coordinates?

Like previously with my code in the prntscreens, if the screen is uncalibrated I can get closer to the edge (go past it for some edges), but if calibrated via settings I will generally not be able to get close to all edges.

So maybe you could just try recalibrating? It's entirely likely you just didn't press right on the coordinates, and so the screen is now slightly uncalibrated so you can't reach the RHS

I tried recalibrating via settings some times also deliberately hitting the crosshairs off center to try different stuff out. It was hit and miss I would say and I had a hard time getting something pretty good with regards to edges.

However I tried the Touchscreen Calibration app from the app loader now again, and I seem to get a better calibration that way at least with regards to accessing edge pixels. It's possible it doesn't calibrate as good in the y-direction well in on the screen where it might register a little below where I intend to touch.

EDIT: It might be that the Touchscreen Calibration app doesn't do anything, so the results are just the same as with an uncalibrated screen.

[thyttan]

The solution then I guess is to have sufficiently large borders where input is not needed for the apps functionality. These could be hardcoded or depend on the users touch screen calibration.

Another solution can be to remove the touch screen calibration on apps where access to the edges is needed and restore it when the app is closed.

[gfwilliams]

As on https://github.com/espruino/BangleApps/pull/2558 - I don't think the solution is to remove touch calibration as for most users this is needed to make the keyboard usable.

As far as I can tell, in kbtouch the default is to use dragging to select the right key, so actually the fact you can't get to the edge doesn't matter at all? It's only if you deliberately set it up with 'one to one' and you calibrate the screen and only then on your watch - so I don't think that's really such a big deal?

But for dragboard it seems we could pretty trivially increase the padding so it works fine? I think that is worth doing.

But it's weird the watch can register closer to the edge when entering and not be able to access the same pixels when draging back off the same edge?

I think this is probably just how the touch controller tries to work out where the finger is - it's effectively got a bitmap of capacitance and it's trying to guess a central point for it. Probably when you have dragged on and it knows how big your finger is, it's more willing to say it's offscreen when only half of it is. Going on, it doesn't know if it's looking at a small or a big finger.

back to what you mentioned on #2558:

Looking at your calibration, the values are way off what I would expect. The defaults (uncalibrated) are 0,0,160,160 and you are 6,9,193,176 - so it is very much as if everything is shifted.

I know you've got your code that draws the X/Y lines - if you update that for every drag event, do those lines really appear to line up with where your finger is when it's calibrated?

I guess the other option is for the calibration to deliberately 'zoom' the coordinates slightly, to ensure that even if calibration was off, you could still reach closer to the edges.

[thyttan]

I know you've got your code that draws the X/Y lines - if you update that for every drag event, do those lines really appear to line up with where your finger is when it's calibrated?

Here's on image showing a fingerprint where my finger was pressed down lightly and the lines present (same calibration as in https://github.com/espruino/BangleApps/pull/2558#issuecomment-1419289597):

Here's on image showing a fingerprint where my finger was pressed down pretty hard and the lines present (same calibration as in https://github.com/espruino/BangleApps/pull/2558#issuecomment-1419289597):

Evaluating my feeling, I'd say it feels just about right in the LR-direction but like the horizontal line should register slightly higher up to be perfect for where it feels like I'm touching.

I guess the other option is for the calibration to deliberately 'zoom' the coordinates slightly, to ensure that even if calibration was off, you could still reach closer to the edges.

Interesting. I don't have an intuition for how that would work/feel so it's hard for me to judge if it's worth putting time into doing that. But certainly interesting!

[thyttan]

As far as I can tell, in kbtouch the default is to use dragging to select the right key, so actually the fact you can't get to the edge doesn't matter at all? It's only if you deliberately set it up with 'one to one' and you calibrate the screen and only then on your watch - so I don't think that's really such a big deal?

No, it's probably not a very big deal, agreed. The one-to-one input is my preferred way of using kbtouch, but probably not many other use it. And I jump between keyboards relatively frequently.

But for dragboard it seems we could pretty trivially increase the padding so it works fine? I think that is worth doing.

I'll probably try that :+1: (EDIT: Although the letters are already quite small)

[gfwilliams]

Thanks for those pics. What happens if you try pressing towards the bottom right of the screen?

Maybe try:

Bangle.on('drag',e => {
  g.clear(1).drawLine(0,e.y, 180,e.y).drawLine(e.x,0,e.x,180);
});

and then just try with dragging your finger?

Those two presses are relatively central on the screen but even so it looks like the one slightly to the bottom right is off center.

I just tried here after calibration and mine feels a bit 'off' too. I think it's the calibration that's broken. I just tried adding a mild 'zoom' to it and it does seem better.

Try this:

// disable touchscreen calibration (passed coords right through)
Bangle.setOptions({touchX1: 0, touchY1: 0, touchX2: g.getWidth(), touchY2: g.getHeight() });

var P = 32;
var corners = [
  [P,P],
  [g.getWidth()-P,P],
  [g.getWidth()-P,g.getHeight()-P],
  [P,g.getHeight()-P],
];
var currentCorner = 0;
var currentTry = 0;
var pt = {
  x1 : 0, y1 : 0, x2 : 0, y2 : 0
};

function showTapSpot() {
  var spot = corners[currentCorner];
  g.clear(1);
  g.drawLine(spot[0]-32,spot[1],spot[0]+32,spot[1]);
  g.drawLine(spot[0],spot[1]-32,spot[0],spot[1]+32);
  g.drawCircle(spot[0],spot[1], 16);
  var tapsLeft = (1-currentTry)*4+(4-currentCorner);
  g.setFont("6x8:2").setFontAlign(0,0).drawString(tapsLeft+/*LANG*/" taps\nto go", g.getWidth()/2, g.getHeight()/2);
}

function calcCalibration() {
  g.clear(1);
  // we should now have 4 of each tap in 'pt'
  pt.x1 /= 4;
  pt.y1 /= 4;
  pt.x2 /= 4;
  pt.y2 /= 4;
  print(pt);
  // work out final values
  var calib = {
    x1 : Math.round(pt.x1 - (pt.x2-pt.x1)*P/(g.getWidth()-P*2)) + 16,
    y1 : Math.round(pt.y1 - (pt.y2-pt.y1)*P/(g.getHeight()-P*2)) + 16,
    x2 : Math.round(pt.x2 + (pt.x2-pt.x1)*P/(g.getWidth()-P*2)) - 16,
    y2 : Math.round(pt.y2 + (pt.y2-pt.y1)*P/(g.getHeight()-P*2)) - 16
  };
  print(calib);
  Bangle.setOptions({
    touchX1: calib.x1, touchY1: calib.y1, touchX2: calib.x2, touchY2: calib.y2
  });
  g.setFont("6x8:2").setFontAlign(0,0).drawString(/*LANG*/"Calibrated!", g.getWidth()/2, g.getHeight()/2);

  Bangle.on('drag',e => {
    g.clear(1).drawLine(0,e.y, 180,e.y).drawLine(e.x,0,e.x,180);
  });
}

function touchHandler(_,e) {
  var spot = corners[currentCorner];
  // store averages
  if (spot[0]*2 < g.getWidth())
    pt.x1 += e.x;
  else
    pt.x2 += e.x;
  if (spot[1]*2 < g.getHeight())
    pt.y1 += e.y;
  else
    pt.y2 += e.y;
  // go to next corner
  currentCorner++;
  if (currentCorner>=corners.length) {
    currentCorner = 0;
    currentTry++;
    if (currentTry==2) {
      Bangle.removeListener('drag', dragHandler);
      return calcCalibration();
    }
  }
  showTapSpot();
}

var dragx=0, dragy=0, dragc=0;
function dragHandler(e) {
  if (e.b) {
    dragx+=e.x;
    dragy+=e.y;
    dragc++;
  } else {
    if (dragc) touchHandler(null, { x : dragx/dragc, y : dragy/dragc});
    dragx=0, dragy=0, dragc=0;
  }
}
Bangle.on('drag', dragHandler);

showTapSpot();

It doesn't save calibration but after it's done you can then press and drag to get an idea of the positioning...

[thyttan]

Thanks for those pics. What happens if you try pressing towards the bottom right of the screen?

Maybe try: (...)

I feel like the accuracy of the touches and drags is pretty much the same for the area I can access when calibrated. I still can't access the edges with that short test code, as before.

I just tried here after calibration and mine feels a bit 'off' too. I think it's the calibration that's broken. I just tried adding a mild 'zoom' to it and it does seem better.

Try this: (...)

With this zoomed calibration I can access virtually the whole screen area (I'm just a few pixels away from bottom and right border). But for me it comes at the expense of accuracy on the area I could reach also with the old calibration. I feel this is to be expected with adding the zoom step? However, it may prove to be a good compromise!

I think this is a bit hard to capture with a video, but if you would want one maybe I can manage to shoot one with a tripod or something :P

[thyttan]

Quoting myself:

I feel this is to be expected with adding the zoom step? However, it may prove to be a good compromise!

I guess some clever math could zoom mainly "towards the bottom right" and not towards top left, and that may be the best thing for my watch and my way of pressing/draging on it. But that is probably overkill :P

[thyttan]

I finally took a stab at this and now have a solution that works with my watch and calibration:

{
  let zoomX = g.getWidth()/153; // 153 was the highest atainable x value after calibrating the touchscreen. Using this to zoom pixel 153 up to 175.
  let zoomY = g.getHeight()/153; // 153 was the highest atainable y value after calibrating the touchscreen. Using this to zoom pixel 153 up to 175.


  let zoomDrag = (e)=>{
    if (e.zoomed) return;
    E.stopEventPropagation();
    e.zoomed = true;
    e.x = Math.round(e.x*zoomX);
    e.y = Math.round(e.y*zoomY);
    Bangle.emit("drag", e);
  };

  Bangle.prependListener("drag", zoomDrag);
}

I add this at the end of the boot code scripts by specifying a high boot number.

With this I can now access virtually all 176x176 pixels also when I've calibrated the screen.

I don't know how well it would work for other people. It's currently not a truly general solution.

Edits:

1. I'll compare it with Gordons code from above to see if I just did what he did or if I can learn something.
2. Comparing the two mine works better on my watch especially with regards to accuracy.
3. Refactor code to depend on user gathered max values instead of looking at Bangle.getOptions()

[gfwilliams]

I'm a little iffy about hard-coded values - while it may work for you it could do with a bunch of testing to see if it's better for everyone.

Also it feels like if some other code did Bangle.prependListener before yours, it'd get called twice, once with zoomed and once without zoomed coordinates - which if it's using drag for scrolling could make scroll twice as fast?

I guess part of me is thinking this issue is 18 months old, and you're the only one to comment on it - so I wonder if it's worth putting too much work into solving this for everyone, as it might be something that's only really an issue for you. If it works for you that's great, and I guess if we can make it an app that can just be installed then that makes it easy for everyone, but I don't think this is something we should be trying to build in yet.

[thyttan]

Makes total sense :)

If I get to a more general solution that still works well for me, then I'll do a PR for an app.

[thyttan]

I realized that applying the zoom affects the accuracy improvements of the previous calibration. So maybe it makes sense to avoid zooming for most of the screen area.

Below is another solution that only applies magnification near the edges I can't reach otherwise (right and bottom). I haven't decided on which solution I like best.

{
  let zoomDrag = (e)=>{
    "ram";
    if (e.zoomed) return;
    E.stopEventPropagation();
    e.zoomed = true;

    // Reachable pixels found by printing the drag event to console. Change to fit your device and calibration.
    let reachableX = 153;
    let startZoomX = 130;
    let reachableY = 153;
    let startZoomY = 130;

    if (e.x >= startZoomX){
      e.x = startZoomX + Math.round((e.x-startZoomX)*(175-startZoomX)/(reachableX-startZoomX));
    }
    if (e.y >= startZoomY){
      e.y = startZoomY + Math.round((e.y-startZoomY)*(175-startZoomY)/(reachableY-startZoomY));
    }

    Bangle.emit("drag", e);
  };

Bangle.prependListener("drag", zoomDrag);
}

[thyttan]

Also it feels like if some other code did Bangle.prependListener before yours, it'd get called twice, once with zoomed and once without zoomed coordinates - which if it's using drag for scrolling could make scroll twice as fast?

Is it possible to listen for prependListener registrations? If so I could listen to such registrations and reinitialize my zoomDrag handler prepended again. Making sure it's always at the front.

But I guess this would open up for infinite loops of re-registering listeners if two or more places in code were to do this.

Edit: I should be able to modify Bangle.prependListener to emit an event I guess.

[gfwilliams]

Is it possible to listen for prependListener registrations?

Yes, i guess you could override prependListener on Bangle.js.

However this all got me thinking... We actually do all this zooming stuff as part of the calibration already: https://github.com/espruino/Espruino/blob/master/libs/banglejs/jswrap_bangle.c#L1992-L1993

So you don't need any of this. Literally all you need is after the touchscreen is calibrated, you take the touchX1/etc values from settings, and you move them slightly closer together - like add 10px to touch1 and take 10px from touch2.

Then it's sorted - so actually your boot code could just be:

var o = Bangle.getOptions();
Bangle.setOptions({
  touchX1:o.touchX1+10,
  touchY1:o.touchY1+10,
  touchX2:o.touchX2-10,
  touchY2:o.touchY2-10});

... but actually I look back now and this is what I was basically suggesting in https://github.com/espruino/BangleApps/issues/2457#issuecomment-1422327622 anyway (tweaking the calibration)...

So basically we 100% do not want something that overrides drag, etc. We have the calibration code for exactly this kind of thing - it's just figuring out what numbers we should use for calibration

[thyttan]

Ok, thanks for the links and example! I'll try a variation of that for a while.

[thyttan]

Just leaving a note.

So what's working best for me is my solution where I manually determine how far I can reach and then intercept and modify the drag event (https://github.com/espruino/BangleApps/issues/2457#issuecomment-2351536856).

So basically we 100% do not want something that overrides drag, etc. We have the calibration code for exactly this kind of thing - it's just figuring out what numbers we should use for calibration

I guess my watch (and/or way of using it) is so special that there is no reason to tweak the way it's done for everyone else. If someone else shows up here saying they have similar troubles - maybe we reconsider then.

[gfwilliams]

what's working best for me is my solution where I ... modify the drag event

Ok, it would be interesting to know why really - what doesn't changing the options do right?

Because changing the options should provide you with better per-pixel accuracy.

I looks like the way you're doing it, you're not cropping the pixel x/y to the screen bounds - could that be it? I'm pretty sure I didn't used to do that in Bangle.js either, but then someone complained so I had to start cropping - so just be aware that some apps may have issues with out of bounds taps.

[thyttan]

what's working best for me is my solution where I ... modify the drag event

Ok, it would be interesting to know why really - what doesn't changing the options do right?

Because changing the options should provide you with better per-pixel accuracy.

It felt like the accuracy decreased on the parts of the screen that already did work with calibration before (approximately the square 0<=x<=153, 0<=y<=153). So I'd rather keep that calibration on most of the screen. But then take some of the rightmost and bottom parts of the reachable area (130<=x<=153 || 130<=y<=153) and stretch them out so I can reach the borders.

I looks like the way you're doing it, you're not cropping the pixel x/y to the screen bounds - could that be it?

I'm not entirely sure what you mean. My solution doesn't let me go beyond the LCD pixels. I take x-values [130, ... , 153] and transform them to [130, ... , 175] (and the same for y-values).

I'm pretty sure I didn't used to do that in Bangle.js either, but then someone complained so I had to start cropping - so just be aware that some apps may have issues with out of bounds taps.

As I say I'm not sure what you mean here. But I wonder if undoing that cropping would help my situation?

My assumption for the ground problem is that the LCD pixels and touch pixels are too misaligned on my watch - and that there are not enough touch pixels to the right and bottom of the screen to use when having done the calibration.

[gfwilliams]

Ahh, ok, gotcha - that makes a lot more sense. So the issue is really that your touchscreen appears nonlinear?

I don't know if this helps but I re-read what you write at the start, about your calibration values: touchX1: -2, touchY1: 9, touchX2: 195, touchY2: 177

The default is 160 for touchX/Y2, so the calibration will absolutely be making it difficult to reach the right and bottom edges (although at least it won't crop the values!). I guess calibration got the weird values precisely because there's something nonlinear with your touchscreen.

So personally, I think this is one of those things where it's probably just your device that needs this - so I'll close for now. If you want to make it an app that's great, but it should make it clear in the description that the first port of call should be just using the normal calibration function.

[thyttan]

Thanks for helping me out 😊