Add ability to get full bounds of an element

[MrGVSV]

Problem

The bounds attribute only returns the visible bounds of an element. There should be a way to either get the visible bounds and the full bounds of an element.

It also seems like this is what's returned by driver.getElementRect() as well.

Solution

Add an attribute that allows users to get the full bounds. Ideally, this would be bounds and visibleBounds. If compatability/breakage is a concern, though, it could also be bounds and fullBounds.

Use Case

My own use-case for this is to check whether a particular edge of an element is fully scrolled to when only part of the element is currently displayed.

This is straightforward on iOS as I can just use the element's frame to calculate the additional amount needed to be scrolled. However, on Android, attempting to calculate the offscreen edge using its bounds will result in the calculation incorrectly indicating it is fully scrolled to (as the height becomes element's top edge to where it's cut off by the scroll view).

Specs

appium: 2.0.0-beta.40 appium-uiautomator2-driver: 2.2.0

[mykola-mokhnach]

Have you tried simpleBoundsCalculation setting ?

[MrGVSV]

Have you tried simpleBoundsCalculation setting ?

Unfortunately, it's still returning only the visible frame.

[MrGVSV]

I found this chunk of code, which I think might be related?

https://github.com/appium/appium-uiautomator2-server/blob/e3ce25db6cdbc8032afe39232cc62d58ad2c1bb5/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoHelper.java#L235-L280

[mykola-mokhnach]

Yes it is related. As far as I can see https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo only provides a single method for fetching element's bounds: https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#getBoundsInScreen(android.graphics.Rect)

[MrGVSV]

Yes it is related. As far as I can see https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo only provides a single method for fetching element's bounds: https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#getBoundsInScreen(android.graphics.Rect)

Does this return the full bounds? If it does, can it be used directly for getting the full bounds of the element and then use the custom getBounds method above for getting the visible bounds?

[mykola-mokhnach]

It returns exactly what's described in the documentation. The above setting makes the server to return the raw result of the getBoundsInScreen method: https://github.com/appium/appium-uiautomator2-server/blob/2abe64df5ba74e9fcb4a4fa3fc0df8ac2e40c484/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoHelper.java#L213

[MrGVSV]

Ah okay I see now. Yeah after looking at the documentation and running a few tests, it seems it does not account for any parts of an element that are offscreen. An element halfway off the bottom edge of the screen results in half the height being returned.

I don't suppose there's a way to get the full bounds apart from the A11yNodeInfo? 😅

[MrGVSV]

I suppose if we had access to the UiObject, we could get this information, right? At least according to these docs (though, I'm not sure if this just returns the same on-screen bounds or not).

We could maybe extract that method here, but that might be too inconsistent/complex:

https://github.com/appium/appium-uiautomator2-server/blob/2abe64df5ba74e9fcb4a4fa3fc0df8ac2e40c484/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoExtractor.java#L56-L66

[mykola-mokhnach]

You could try to experiment locally and see what it returns. Maybe there is still something of use. This is the actual code of getBounds and getVisibleBounds methods from UIAutomator framework:

    /**
     * Returns the view's <code>bounds</code> property. See {@link #getVisibleBounds()}
     *
     * @return Rect
     * @throws UiObjectNotFoundException
     * @since API Level 16
     */
    public Rect getBounds() throws UiObjectNotFoundException {
        Tracer.trace();
        AccessibilityNodeInfo node = findAccessibilityNodeInfo(mConfig.getWaitForSelectorTimeout());
        if(node == null) {
            throw new UiObjectNotFoundException(mUiSelector.toString());
        }
        Rect nodeRect = new Rect();
        node.getBoundsInScreen(nodeRect);

        return nodeRect;
    }

    /**
     * Finds the visible bounds of a partially visible UI element
     *
     * @param node
     * @return null if node is null, else a Rect containing visible bounds
     */
    private Rect getVisibleBounds(AccessibilityNodeInfo node) {
        if (node == null) {
            return null;
        }

        // targeted node's bounds
        int w = getDevice().getDisplayWidth();
        int h = getDevice().getDisplayHeight();
        Rect nodeRect = AccessibilityNodeInfoHelper.getVisibleBoundsInScreen(node, w, h);

        // is the targeted node within a scrollable container?
        AccessibilityNodeInfo scrollableParentNode = getScrollableParent(node);
        if(scrollableParentNode == null) {
            // nothing to adjust for so return the node's Rect as is
            return nodeRect;
        }

        // Scrollable parent's visible bounds
        Rect parentRect = AccessibilityNodeInfoHelper
                .getVisibleBoundsInScreen(scrollableParentNode, w, h);
        // adjust for partial clipping of targeted by parent node if required
        nodeRect.intersect(parentRect);
        return nodeRect;
    }

[jlipps]

it seems it does not account for any parts of an element that are offscreen

this is a fact about working with android screens that I've noticed, anything not on the screen doesn't exist.