[BUG] Additional bug affecting predicates on preceding and following steps

[joewiz]

Describe the bug

@line-o's PR https://github.com/eXist-db/exist/pull/4108 successfully fixed the cases provided in https://github.com/eXist-db/exist/issues/4085 - i.e., where the filter exercised was simply [true()] - but further testing has revealed additional cases that trigger the same bug. For example, if [true()] is changed to one that selects the context node, such as [exists(.)], even a patched system will produce the same failures as described in the original issue.

Here, too, eXist fails to locate some nodes along the preceding and following axes, when a document is stored in the database and when the predicate described has been applied to the for clause of a FLWOR expression or to the LHS of a simple map operator.

Expected behavior

eXist should locate all matching nodes regardless of the XPath axis used and the expression used in a predicate that filter the result set.

To reproduce

The sample scenario and XQSuite test shown here is identical to the original XQSuite in https://github.com/eXist-db/exist/issues/4085, except the predicate is changed to one that selects the context node. With this change, even a system patched with https://github.com/eXist-db/exist/pull/4108 will produce errors identical to the original report.

As before, the bug arises only when the document stored in the database, and when a predicate is applied to the initial expression.

For example, given an XML document, stored in eXist as /db/test/test.xml:

<root>
    <pb id="pb1"/>
    <div>
        <p>
            <w id="w1"/>
            <w id="w2"/> 
            <w id="w3"/>
        </p>
        <pb id="pb2"/>
        <p>
            <w id="w4"/> 
            <w id="w5"/>
        </p>
    </div>
    <pb id="pb3"/>
</root>

... a query that iterates over this document's <w> elements and applies a filter that selects the context node - even a meaningless [exists(.)] - and then traverses the preceding axis to select the element's preceding <pb> element, will trigger the bug. Here's an example query:

for $w in doc("/db/test/test.xml")//w[exists(.)]
return
    $w/preceding::pb[1]

w/@id	expected result	actual result
w1	<pb id="pb1"/>	<pb id="pb1"/>
w2	<pb id="pb1"/>	()
w3	<pb id="pb1"/>	()
w4	<pb id="pb2"/>	<pb id="pb2"/>
w5	<pb id="pb2"/>	()

The XQSuite below performs this and variations on the test—showing the bug affects not just FLWOR expressions but simple map operator expressions too, and not just the preceding but also the following axis. The corresponding failing tests are:

• preceding-with-predicate-db-flwor
• preceding-with-predicate-db-map
• following-with-predicate-db-flwor
• following-with-predicate-db-map

xquery version "3.1";

module namespace t="http://exist-db.org/xquery/test";

declare namespace test="http://exist-db.org/xquery/xqsuite";

declare variable $t:XML := document {
<root>
    <pb id="pb1"/>
    <div>
        <p>
            <w id="w1"/>
            <w id="w2"/> 
            <w id="w3"/>
        </p>
        <pb id="pb2"/>
        <p>
            <w id="w4"/> 
            <w id="w5"/>
        </p>
    </div>
    <pb id="pb3"/>
</root>
};

declare
    %test:setUp
function t:setup() {
    let $testCol := xmldb:create-collection("/db", "test")
    return
        xmldb:store("/db/test", "test.xml", $t:XML)
};

declare
    %test:tearDown
function t:tearDown() {
    xmldb:remove("/db/test")
};

(: PRECEDING AXIS TESTS :)

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-with-predicate-mem-flwor() {
    for $w in $t:XML//w[exists(.)]
    let $preceding-page := $w/preceding::pb[1]
    return
        if ($preceding-page) then
            $w/@id || ":" || $preceding-page/@id
        else
            $w/@id || ":PRECEDING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-with-predicate-mem-map() {
    $t:XML//w[exists(.)] 
        ! (./@id || ":" || (./preceding::pb[1]/@id, "PRECEDING_PB_NOT_FOUND")[1])
};

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-with-predicate-db-flwor() {
    for $w in doc("/db/test/test.xml")//w[exists(.)]
    let $preceding-page := $w/preceding::pb[1]
    return
        if ($preceding-page) then
            $w/@id || ":" || $preceding-page/@id
        else
            $w/@id || ":PRECEDING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-with-predicate-db-map() {
    doc("/db/test/test.xml")//w[exists(.)] 
        ! (./@id || ":" || (./preceding::pb[1]/@id, "PRECEDING_PB_NOT_FOUND")[1])
};

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-without-predicate-flwor() {
    for $w in doc("/db/test/test.xml")//w
    let $preceding-page := $w/preceding::pb[1]
    return
        if ($preceding-page) then
            $w/@id || ":" || $preceding-page/@id
        else
            $w/@id || ":PRECEDING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb1", "w2:pb1", "w3:pb1", "w4:pb2", "w5:pb2")
function t:preceding-without-predicate-map() {
    doc("/db/test/test.xml")//w 
        ! (./@id || ":" || (./preceding::pb[1]/@id, "PRECEDING_PB_NOT_FOUND")[1])
};

(: FOLLOWING AXIS TESTS :)

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-with-predicate-mem-flwor() {
    for $w in $t:XML//w[exists(.)]
    let $following-page := $w/following::pb[1]
    return
        if ($following-page) then
            $w/@id || ":" || $following-page/@id
        else
            $w/@id || ":FOLLOWING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-with-predicate-mem-map() {
    $t:XML//w[exists(.)] 
        ! (./@id || ":" || (./following::pb[1]/@id, "FOLLOWING_PB_NOT_FOUND")[1])
};

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-with-predicate-db-flwor() {
    for $w in doc("/db/test/test.xml")//w[exists(.)]
    let $following-page := $w/following::pb[1]
    return
        if ($following-page) then
            $w/@id || ":" || $following-page/@id
        else
            $w/@id || ":FOLLOWING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-with-predicate-db-map() {
    doc("/db/test/test.xml")//w[exists(.)] 
        ! (./@id || ":" || (./following::pb[1]/@id, "FOLLOWING_PB_NOT_FOUND")[1])
};

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-without-predicate-flwor() {
    for $w in doc("/db/test/test.xml")//w
    let $following-page := $w/following::pb[1]
    return
        if ($following-page) then
            $w/@id || ":" || $following-page/@id
        else
            $w/@id || ":FOLLOWING_PB_NOT_FOUND"
};

declare
    %test:assertEquals("w1:pb2", "w2:pb2", "w3:pb2", "w4:pb3", "w5:pb3")
function t:following-without-predicate-map() {
    doc("/db/test/test.xml")//w 
        ! (./@id || ":" || (./following::pb[1]/@id, "FOLLOWING_PB_NOT_FOUND")[1])
};

The results of this test are as follows:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite package="http://exist-db.org/xquery/test" timestamp="2021-12-08T12:52:18.256-05:00"
    tests="12" failures="4" errors="0" pending="0" time="PT0.009S">
    <testcase name="following-with-predicate-db-flwor" class="t:following-with-predicate-db-flwor">
        <failure message="assertEquals failed." type="failure-error-code-1">w1:pb2 w2:pb2 w3:pb2
            w4:pb3 w5:pb3</failure>
        <output>w1:pb2 w2:FOLLOWING_PB_NOT_FOUND w3:FOLLOWING_PB_NOT_FOUND w4:pb3
            w5:FOLLOWING_PB_NOT_FOUND</output>
    </testcase>
    <testcase name="following-with-predicate-db-map" class="t:following-with-predicate-db-map">
        <failure message="assertEquals failed." type="failure-error-code-1">w1:pb2 w2:pb2 w3:pb2
            w4:pb3 w5:pb3</failure>
        <output>w1:pb2 w2:FOLLOWING_PB_NOT_FOUND w3:FOLLOWING_PB_NOT_FOUND w4:pb3
            w5:FOLLOWING_PB_NOT_FOUND</output>
    </testcase>
    <testcase name="following-with-predicate-mem-flwor" class="t:following-with-predicate-mem-flwor"/>
    <testcase name="following-with-predicate-mem-map" class="t:following-with-predicate-mem-map"/>
    <testcase name="following-without-predicate-flwor" class="t:following-without-predicate-flwor"/>
    <testcase name="following-without-predicate-map" class="t:following-without-predicate-map"/>
    <testcase name="preceding-with-predicate-db-flwor" class="t:preceding-with-predicate-db-flwor">
        <failure message="assertEquals failed." type="failure-error-code-1">w1:pb1 w2:pb1 w3:pb1
            w4:pb2 w5:pb2</failure>
        <output>w1:pb1 w2:PRECEDING_PB_NOT_FOUND w3:PRECEDING_PB_NOT_FOUND w4:pb2
            w5:PRECEDING_PB_NOT_FOUND</output>
    </testcase>
    <testcase name="preceding-with-predicate-db-map" class="t:preceding-with-predicate-db-map">
        <failure message="assertEquals failed." type="failure-error-code-1">w1:pb1 w2:pb1 w3:pb1
            w4:pb2 w5:pb2</failure>
        <output>w1:pb1 w2:PRECEDING_PB_NOT_FOUND w3:PRECEDING_PB_NOT_FOUND w4:pb2
            w5:PRECEDING_PB_NOT_FOUND</output>
    </testcase>
    <testcase name="preceding-with-predicate-mem-flwor" class="t:preceding-with-predicate-mem-flwor"/>
    <testcase name="preceding-with-predicate-mem-map" class="t:preceding-with-predicate-mem-map"/>
    <testcase name="preceding-without-predicate-flwor" class="t:preceding-without-predicate-flwor"/>
    <testcase name="preceding-without-predicate-map" class="t:preceding-without-predicate-map"/>
</testsuite>

Context (please always complete the following information):

• OS: macOS 12.0.1
• eXist-db version: eXist 5.4.0-SNAPSHOT 8cc05b03d68b0b1b5460327a0d9f4d654987cc17 20211208074308 (built from https://github.com/eXist-db/exist/pull/4108)
• Java Version: OpenJDK 1.8.0_312 (liberica-jdk8-full 1.8.0_312-b07)

Additional context

• How is eXist-db installed? 5.4.0-SNAPSHOT built from source, run as Mac app
• Any custom changes in e.g. conf.xml? none

[line-o]

If the predicate does not (explicitly) depend on the context item we receive the expected return values.

doc("/db/test/test.xml")//w[exists(self::w)]  ! preceding::pb

[joewiz]

@line-o At first, I wondered if, in the same way that your original fix identified a problem with the abbreviated syntax // that didn't affect the verbose equivalent /descendant-or-self::node()/, could it be that this bug affects primarily the abbreviated syntax . but doesn't not the verbose equivalent self::node()? However, the following formulations exhibit the same error:

doc("/db/test/test.xml")//w[exists(self::node())]
doc("/db/test/test.xml")//w[exists(self::element())]

... though these formulations return the expected results:

doc("/db/test/test.xml")//w[exists(self::w)]
doc("/db/test/test.xml")//w[exists(self::element(w))]

Not sure if this helps shed any light on the issue...