[BUG in Expression] Null or incorrect expression field in JSON output after delete operation

[iscanegemen]

Bug Report

Applying the delete operation while updating the table creates either incorrect or null results on the resulting expression field in the view.to_json() output in JS, when an expression is applied.

We have created a Github repository for demonstrating the problem via two separate sample poc applications. In the both of these applications we have created, there is a Perspective Viewer with the following expression applied: if (match("string", 'B')) {"float" * -1} else {"float"}

Null outcome: https://github.com/kryaksy/perspective-poc/tree/expression-null-issue Incorrect outcome: https://github.com/kryaksy/perspective-poc/tree/expression-equality-issue

Steps to Reproduce:

1. Go to one of the given Github repositories, which demonstrate the null result from the expression and the issue for values that do not match, respectively.
2. Click on the link under the README file to start the application.
3. When the application starts, the delete operation is selected by default. The view applies an update, insert and delete operation periodically. When the program encounters a JSON output, it will stop and display the error message.
4. Inspect the JSON output. Observe that the "exp" value in the JSON output is not correct, according to the expression that is applied.
5. After having seen the error, uncheck the delete button. It will stop the delete operation and the error will be gone.

Expected Result:

We expect the JSON output to have a correct value for the "exp" field. The absolute value of the "exp" and the "float" fields should match, since the expression only multiplies the value of the "float" field with -1 based on a simple condition inside the if statement.

Actual Result:

The JSON output has null values for the "exp" field or values whose absolute value doesn't match with the value of the "float" column.

Examples include the following:

{
    "string": "FD",
    "float": 0.4013222456195562,
    "exp": 0.6984378020751012
}

Also:

{
    "string": "CA",
    "float": 0.6946245288673509,
    "exp": null
}

Environment:

Perspective version is up-to-date.

Additional Context:

We have also noticed that this error does not occur in Perspective-Viewer, just the view.to_json() output has this issue. Also, this does not occur with the insert and update operations.

[iscanegemen]

Hi, are there any updates on this issue? It is blocking an important feature.

[kryaksy]

It seems that our poc wasn't working correctly because of the api changes in version 1.2.0. We've just pushed the needed changes and it works as expected now.

We always spend time to reveal the possible issues as clear as possible since we attach great importance for this project. Can you please check this issue or suggest us some kind of workaround? Of course, let us know if you have any advice to improve our issue reproduction.

Thanks in advance.

[kryaksy]

We realized that the perspective viewer also has the similar issue when configured a group_by. You can see the issue here: https://github.com/kryaksy/perspective-poc/tree/expression-equality-issue-in-viewer

[kryaksy]

Hey @texodus, @sc1f, did you have a chance review this issue?

[kryaksy]

Any update?

[timkpaine]

@kryaksy we'll provide updates as soon as they are available, there's no need to ask multiple times on the same issue for updates. We try to do a good job of linking a PR to corresponding issues, so you'll see when something is nearing completion. Otherwise it's best to assume everything is under consideration or in progress pending other priorities.

[kryaksy]

@timkpaine Sorry to spam this issue. You may be right. I just wanted to learn if this is a bug or we are doing something wrong or any thought about this. Lesson learned. Thanks for the comment.

[timkpaine]

@kryaksy no worries! it's almost certainly a bug there's just lots of stuff we're working on and deletes in particular require us to take a deeper look

[kryaksy]

We realized that the perspective viewer also has the similar issue when configured a group_by. You can see the issue here: https://github.com/kryaksy/perspective-poc/tree/expression-equality-issue-in-viewer

It's worth to mention. This issue seems to be resolved in perspective-viewer in v1.4.0. But still current for the other examples.

[texodus]

I've checked out these repos and they don't do what the bug description says they do (there's no remove(), (not delete() I presume ...), no exp, etc) - rather they seem to just log the JSON output of whatever operations you perform on <perspective-viewer>. A clear repro dramatically reduces the amount of time a developer needs to spend to get up-to-date on an issue, without one I can't really say exactly whether this is a bug or not.

Independent of this, we have had a good repro of a similar-ish sounding bug reported which occurs in the mixed presence of remove() on a group_by context with an expression column. If this is the same issue, it is resolved by #2311. If not, please re-open with the correct repro on v2.3.2