Unclear or incorrect arithmetic update-assignment documentation

[wader]

Hi

I noticed something when i tried to use the current value for +=. In the documentation it says:

jq has a few operators of the form a op= b, which are all equivalent to a |= . op b. So, += 1 can be used to increment values, being the same as |= . + 1.

But it seems like for a op= b the input for the RHS is the input of the LHS and not . | a.

$ jq -n '{a:1} | .a += (debug | 1)'
["DEBUG:",{"a":1}]
{
  "a": 2
}
$ jq -n '{a:1} | .a |= . + (debug | 1)'
["DEBUG:",1]
{
  "a": 2
}

In my case I was trying to do something similar to this. Add something to a list conditionally on the current list value:

$ jq -n '{a:[1,2,3]} | .a += [if length % 3 == 0 then 4 else empty end]'
{
  "a": [
    1,
    2,
    3
  ]
}

Here length will get {a:[1,2,3]} as input not [1,2,3] and return 1. But if i use |= i get what i wanted:

$ jq -n '{a:[1,2,3]} | .a |= . + [if length % 3 == 0 then 4 else empty end]'
{
  "a": [
    1,
    2,
    3,
    4
  ]
}

gojq seems to have the same behaviour which makes me think i might misunderstanding something? if so I can try to improve the documentation. If it's a bug it feels it might be a bit too late to change the behaviour?

[itchyny]

 $ jq -nc '[[0,[1]],[2]] | .[0] += .[1]'
[[0,[1],2],[2]]

 $ jq -nc '[[0,[1]],[2]] | .[0] |= . + .[1]'
[[0,[1],1],[2]]

Yeah, a op= b is not equivalent to a |= . op b. Maybe it's too late. But I feel the both behaviors look intuitively correct and fixing the document is better than breaking things.

[wader]

I noticed now that there is at least one a op= b test that would fail with the documented behavior https://github.com/stedolan/jq/blob/master/tests/jq.test#L1016. I guess the tests are as close to a jq specification as there is?

[pkoppstein]

@wader, @itchyny -

It will probably be a long while before the official documentation is updated, so I was thinking of adding a section to the jq FAQ. How about this:

Q: What is the difference between |= and the other update-assignment operators?

A: There are three types of update-assignment operators:

• the simple one: =
• the fancy one: |=
• all the others - loosely speaking, the arithmetic update-assignment operators

In the following, we will use += as the examplar or holotype of the arithmetic update-assignment operators as the basic principle governing their semantics is the same in all cases except for //=, as noted below.

The update-assignment operators operate on one or more JSON values as input. To simplify things, let's focus on the case when the input is a single value, say V. To explain the differences, then, it suffices to examine the expression:

V | (E op= F)

for the different case of "op=".

In a nutshell:

(0) V | (E = F) is identical to V | (E = (V|F))

(1) V | (E |= F) is identical to V | (E = (V|E|F))

(2) V | (E += F) is identical to V | (E = (V|E) + (V|F))

The caveat is that when evaluating the expression V | E //= F, the current C and Go-based implementations of jq both evaluate F unconditionally, as if by:

V | F as $f || E = (V|E) // $f

Examples:

(0) {"a":1, "b":2} | .a = .b #=> {"a": 2, "b": 2}

(1) {"a": {"b":"B"}} | .a |= .b #=> {"a": "B"}

(2) {"a": 1, "b": 2} | .a += .b #=> {"a": 3, "b": 2}

Executable tests:

def assertEqual(x;y):
  if x == y then empty else [x,y] end;

def t0:
 {"a":1, "b":2}
 | . as $v
 | assertEqual( .a = .b;  .a = ($v|.b));

def t1:
 {"a": {"b":"B"}}
 | . as $v
 | assertEqual(.a |= .b; .a = ($v|.a|.b));

def t2:
 {"a": 1, "b": 2}
 | . as $v
 | assertEqual(.a += .b ; .a = ($v|.a) +($v|.b));

t0,t1,t2

[wader]

@pkoppstein Yes would be a good additions i think. I will probably send PR to update to manual also unless someone does it before me. Hopefully it will get merged at some point.

As some trivia I also noticed for //= the RHS is always evaluated even if LHS is not null/false, but guess one can only noticed it by using something with side effects like input, debug etc.

$ jq -n '{a:1} | .a //= (debug | 2)'
["DEBUG:",{"a":1}]
{
  "a": 1
}
$ jq -n '{b:1} | .a //= (debug | 2)'
["DEBUG:",{"b":1}]
{
  "b": 1,
  "a": 2
}

[pkoppstein]

@wader - It seems to me that the behavior of //= that you describe is incorrect and should be fixed, and that it should not be documented except as a bug.

[wader]

Yes i was not expecting it to evaluate RHS and after thinking a bit more in regards to jq having functions with side effects i agree it feels like a bug. Would it be resonable to use debug to see if RHS was evaluated or not? i think so.

@itchyny gojq seems to have the same behavior, what do you think?

[itchyny]

I didn't aware until now but that behavior looks buggy for sure, it should not evaluate RHS just like null coalescing operators in other languages, like C#, JavaScript, and Perl. Note that jq is kind a functional language with purity (with a few exceptions as you would know), and it sometimes has non-standard evaluation order which many people don't notice. jq evaluates RHS first in LHS + RHS but most people don't aware of this.

[wader]

Extracted the //= behavior into its own issue so it's not forgotten #2410

[wader]

@itchyny I didn't know about the LHS + RHS evaluation order, do you know why that is the case?

[nicowilliams]

@itchyny I didn't know about the LHS + RHS evaluation order, do you know why that is the case?

If I ever knew why, I've forgotten. I think this is a question for @stedolan.