Empty values and column-data-storage refactor

[JorisGoosen]

Empty values should be stored in the data and per column.

This is some work I did on it in the runup to 0.18.2 but felt it would become a too big change.

Putting it here to work on it later this release.

Possible problems: [X] importing from 0.17.3 and earlier (converting from emptyValuesMap in metadata.json) Ignore: importing from 0.18.2 (broken emptyvalues (i might just ignore it)) [X] importing from 0.18.3 (breaking up emptyvalues into per workspace and column) Broken: [] also making sure that we can still import scalar columns with a non-scalar value in them that is not convertible to a double (like in the testcsv tsv ods etc) [] Add locale support (should still be done but testing will take time anyway) [] importing a nominal/ordinal now as scalar means getting the ints we wrote to the _ints column. We should probably try to use the "value" shown in the label-editor instead! :o This will help with https://github.com/jasp-stats/jasp-issues/issues/2530 or https://github.com/jasp-stats/jasp-issues/issues/2325 )

sigh

[JorisGoosen]

[JorisGoosen]

I started a bunch of builds under emptyValuesRefactor in case anybode feels like testing it out.

I think this is probably ready now?

[JorisGoosen]

Its maybe not completely done, but close enough that you guys could start reviewing it. And Ill also ask people to test it once the nightlies are build because im sure there are some things ive missed...

[JorisGoosen]

macos arm macos x86

Aah, windows doesnt work it seems...

[shun2wang]

the build test failed maybe also failing on windows.

[RensDofferhoff]

• Press new data
• add any string in an empty column
• No Type is detected
• Use this column in descriptives and the engine crashes

[JorisGoosen]

• Press new data

  • add any string in an empty column

  • No Type is detected

  • Use this column in descriptives and the engine crashes

Oops!

Edit: shouldve set columntype with the default values... Nightlies are building with this fix for anyone curious

[JorisGoosen]

Aah moving labels also doesnt work anymore for columns where all values are convertible to double

[boutinb]

Some issues:

• The empty values are empty for a column, though the workspace has some empty values
• In Edit mode, if I change a value, and press the tab key, the change is lost
• With a nominal column, if I set a new value, it does not appear in the Label Editor.
• I have tried to add a computed column with just contNormal + 1: the engine crashes

Nice feature to see the labels for a scale column (if they are not the same as the value). It could be also nice to see the value of nominal or ordinal column, if the label and the value is not the same.

[boutinb]

If you set a column to use custom empty values, delete all empty values, and save the JASP file, then when reloading the JASP file, the 'Use custom values' for this column is unset. This is because this setting is not saved in the database, and you just try to find out whether a column uses custom empty values by checking whether it has some empty values. The settings 'Use custom values' must be saved in the database.

[JorisGoosen]

Ooh yeah oops

[boutinb]

If you add a new column, you don't get the Label Editor in the Variables Window.

[boutinb]

In the Label Editor, if the Label and Value have the same value, when I change the Value I expect the Label to get automatically the same value. Also if you change the label of the debinf column (in debug.csv), it crashes in column.cpp line 1234, because apparently the ColumnUtils::getDoubleValue fails.

[JorisGoosen]

In the Label Editor, if the Label and Value have the same value, when I change the Value I expect the Label to get automatically the same value.

Uhm, why? That would mess up recoding quite a bit... Suppose youve got:

a	| a
b	| b
c	| c

And you want to recode it to:

1	| a
2	| b
3	| c

Then your proposed change would make that quite annoying

[shun2wang]

In the Label Editor, if the Label and Value have the same value, when I change the Value I expect the Label to get automatically the same value.

well, I think so. because change a old value means "new value", so new value's label should be same value showing in cell. I guess in SPSS it's also same behavior, cells will showing value itself in this situation.

[boutinb]

In the Label Editor, if the Label and Value have the same value, when I change the Value I expect the Label to get automatically the same value.

Uhm, why? That would mess up recoding quite a bit... Suppose youve got:

a	| a
b	| b
c	| c

And you want to recode it to:

1	| a
2	| b
3	| c

Then your proposed change would make that quite annoying

I think a common case would be that if the values are 1, 2, 3..., then people wants to recode to 0, 1, 2... or 10, 20, 30.... Then by recoding them, you don't see the recoded values in the data panel, because they get their old values as labels. Quite annoying I think

[JorisGoosen]

Maybe I could just replace the label if it is the same integer/double value and otherwise not.

[boutinb]

Maybe one solution would be to have an empty label if the value is an integer or a double value, in this case the displayed value in the data view will be the value, and if a value is changed (recoded), then no label is created. For string values, it makes sense to have immediately labels.

[JorisGoosen]

Yeah but then once youve gone to a string label you cant go back anyway, so its the same thing.

[boutinb]

When several columns are selected, it would be nice to have the background color of the headers of all selected columns to be darker. Now only one column header is darker:

[JorisGoosen]

Ive fixed the value changing when the label is the same and a number.

darker color header selected columns

Good idea

Also there are still some things wrong with the labeleditor dissappearing once there are no more labels and then not coming back.

But at least for debInf you now dont get a crash. But if you put ∞ as a missing value you can make the labeleditor dissappeart from it and it wont come back :p

[JorisGoosen]

New nightlies are building but I guess the windows one will fail again...

[boutinb]

Also when the missing values are not custom for a column, you don't see which missing values will be used. It would be nice to see the workspace empty values in grey in place of a empty list:

[boutinb]

Nice feature to see the labels for a scale column (if they are not the same as the value). It could be also nice to see the value of nominal or ordinal column, if the label and the value is not the same.

Don't you think that would be nice also? In fact I think I would always set the Label as displayed value when exists, and the original value in grey (if not the same as the label)

[boutinb]

If you open the Label Editor for a nominal column, and change one label: if you undo it, then is will undo many dummy things like "undo set name from 'contBinom' to 'contBinom'". Then it will really undo the label change. If you then redo it, it will crash (or sometimes add an extra label). This was working in. 0.18.3

[shun2wang]

New nightlies are building but I guess the windows one will fail again...

I can build it local on Windows but Linux still failing.

[JorisGoosen]

Nice feature to see the labels for a scale column (if they are not the same as the value). It could be also nice to see the value of nominal or ordinal column, if the label and the value is not the same.

Don't you think that would be nice also?

Maybe, it might just make the interface a bit cluttered. If you need to see the values then you should switch to scalar.

In fact I think I would always set the Label as displayed value when exists, and the original value in grey (if not the same as the label)

I disagree, because the data you get for a scalar column is the values and not the labels, while for ordinal/nominal its the labels. And this is also why I think the value shouldnt be shown for ordinal/nominal.

[JorisGoosen]

@RensDofferhoff @shun2wang I think linux should now be working?

[shun2wang]

Yes, it can be build on my local now.

[JorisGoosen]

Hmm, cut-copy-pasting with labels different from the values (see attached file, rename to .sav) SPPS_Debug.zip