Implement double bar (||) operator for partial string lists

[jjtolton]

Implements the double bar (||) operator as specified in: https://www.complang.tuwien.ac.at/ulrich/iso-prolog/double_bar

as discussed in https://github.com/mthom/scryer-prolog/pull/3132#issuecomment-3435777584

Summary

Adds support for combining double-quoted strings with partial list notation:

• "abc"||K produces [a,b,c|K]
• "a"||"b"||"c" produces [a,b,c]
• ""||K unifies with K (empty collapse)
• Multi-line support with comments
• Proper syntax validation

Changes

Lexer (src/parser/lexer.rs)

• Added DoubleBar token type
• Modified lexer to detect || (vs single |)

Parser (src/parser/parser.rs)

• Added DoubleBar to TokenType enum
• Implemented operator handling with priority 1 (as per spec)
• Validates that || only appears after string literals:
  • ✅ Allows: "abc"||K
  • ❌ Rejects: K||[] (variable)
  • ❌ Rejects: ("a")||[] (parenthesized)
• Handles all edge cases:
  • Non-empty strings: Creates PartialString term
  • Code lists: Replaces tail in cons cells
  • Empty strings: Correctly collapse to tail

Tests (src/tests/double_bar.pl)

Comprehensive Prolog integration tests (11 tests, all passing):

• Basic functionality
• Multi-line with line comments
• Multi-line with block comments
• Empty string edge cases
• Chaining multiple strings
• All invalid syntax cases

Examples

?- L = "abc"||K.
   L = [a,b,c|K].

?- L = "a"||"b"||"c".
   L = [a,b,c].

?- L = ""||K.
   L = K.

?- L = "a"|| % multi-line
       "b"||
       "c".
   L = [a,b,c].

?- L = K||[].
   error(syntax_error(incomplete_reduction),...).

?- L = ("a")||[].
   error(syntax_error(incomplete_reduction),...).

Testing

All Prolog tests pass:

./target/debug/scryer-prolog -f --no-add-history src/tests/double_bar.pl \
  -f -g "use_module(library(double_bar_tests)), double_bar_tests:main_quiet(double_bar_tests)"

Output: All tests passed

[hurufu]

IMO this can be done using 3 operators: prefix |, suffix | and infix || during term expansion. Similarly to what I did here.

[UWN]

IMO this can be done using 3 operators: prefix |, suffix | and infix || during term expansion. Similarly to what I did here.

1mo, there is no term-expansion during read/1.

2do, it must be done with priority lower than 1. Otherwise user defined operators would interfere.

3tio, [1,2]||L is invalid. Only double quoted lists meaning chars or codes may be used.

4to, functional notation is not an alternative.

5to, '|' must not be a prefix operator. And an operator cannot suffix, just because it is often defined as an infix.

[bakaq]

Note that the proposed syntax for this is much more lax than this, and arbitrary layout chars (whitespace and comments) should be allowed before, between and after each of the bars. The added tests currently only check layout chars after the second bar.

[jjtolton]

Fixed the issue raised by @triska regarding [_]||Rs being incorrectly accepted.

Changes

The parser now correctly rejects list literals before ||:

?- [1,2,3]||K.
   error(syntax_error(incomplete_reduction),...)

?- [_]||Rs.
   error(syntax_error(incomplete_reduction),...)

Implementation

Removed the Term::Cons case from validation - it was accepting ANY list construct. Now only CompleteString and PartialString terms are allowed before ||, ensuring only double-quoted string literals work:

• ✅ "abc"||K - valid string literal
• ❌ [1,2,3]||K - list literal (now rejected)
• ❌ [_]||Rs - list with variable (now rejected)

All existing valid cases still work, and all Prolog tests pass.

[bakaq]

I'm pretty sure that [a,b,c]||S still gets unexpectedly accepted like this, because the [a,b,c] is currently being parsed as a string.

[jjtolton]

I'm pretty sure that [a,b,c]||S still gets unexpectedly accepted like this, because the [a,b,c] is currently being parsed as a string.

So it's syntactically invalid even if it's semantically equivalent to a string? (this indeed is a valid parse under af38843dda191a89de518c9ca4fdf97c2fa46eed)

[bakaq]

Yep, it does accept it:

?- A = [a,b,c]||X.
   A = [a,b,c|X].

So it's syntactically invalid even if it's semantically equivalent to a string?

Yes, the double bar proposal is just syntax, and it should work only with double quotes syntax, and not any other case, as specified in the syntax description term = double quoted list, bar, bar, term ;.

"asdfasd" | | /* a */ A is valid, [a,b,c]||X or even ("asdfa")||X is not.

[jjtolton]

Yep, it does accept it:

?- A = [a,b,c]||X.
   A = [a,b,c|X].

So it's syntactically invalid even if it's semantically equivalent to a string?

Yes, the double bar proposal is just syntax, and it should work only with double quotes syntax, and not any other case, as specified in the syntax description term = double quoted list, bar, bar, term ;.

"asdfasd" | | /* a */ A is valid, [a,b,c]||X or even ("asdfa")||X is not.

ah kkk

[jjtolton]

Yep, it does accept it:

?- A = [a,b,c]||X.

   A = [a,b,c|X].

So it's syntactically invalid even if it's semantically equivalent to a string?

Yes, the double bar proposal is just syntax, and it should work only with double quotes syntax, and not any other case, as specified in the syntax description term = double quoted list, bar, bar, term ;.

"asdfasd" | | /* a */ A is valid, [a,b,c]||X or even ("asdfa")||X is not.

https://github.com/mthom/scryer-prolog/commit/9b04bcc42ce91dd063f0a48ab6fd488ebe07c219

[jjtolton]

Fixed #3160: Empty list [] was incorrectly accepted before ||.

Commit: 16ea9d3d

—J.J.'s robot.