Smaller float types are broken

[philrz]

tl;dr

The float16 and float32 Zed types are broken.

Details

Repro is with Zed commit 91ee553.

Start from this simple math.

$ zq -version
Version: v1.8.1-67-g91ee553a

$ zq 'yield 5 - 4'
1

If I perform this with all combinations of numeric Zed types that are currently implemented, I get bad results for the combinations that mix in the smaller float types like float16 and float32.

$ cat types.txt 
uint8
uint16
uint32
uint64
int8
int16
int32
int64
float16
float32
float64

$ for type1 in $(cat types.txt); do for type2 in $(cat types.txt); do   echo -n "$type1 $type2: " ; echo "{num1: 5($type1), num2: 4($type2)}" | zq -z "yield num1 - num2" -; done; done | egrep -v " 1$"\|" 1\.$"\|" 1\.\("\|" 1\("
float16 float32: -Inf(float16)
float16 float64: -Inf(float16)
float32 float16: -17403.(float32)
float32 float64: -4616189618054758400.(float32)
float64 float16: -17403.
float64 float32: -1082130427.

[philrz]

While fielding an inquiry from a community zync user I became aware that these smaller float types also have a subtle bug when used as join keys. Repro is with Zed commit 38763f8.

Start from this working example. In absence of explicit casting, the num value from the CSV input is stored as a float64 and the num value from the ZSON input is stored as int64. The comparison of the join keys succeeds.

$ zq -version
Version: v1.14.0-16-g38763f82

$ cat f64.csv 
num,word
1,hello

$ cat i64.zson 
{num:1,word:"goodbye"}

$ cat join-1.zed 
file f64.csv
| inner join (
  file i64.zson
) on num=num otherword:=word

$ zq -I join-1.zed 
{num:1.,word:"hello",otherword:"goodbye"}

However, if we make the num value from the ZSON input into a float32, the join fails.

$ cat f32.zson 
{num:1 (float32),word:"goodbye"}

$ cat join-2.zed 
file f64.csv
| inner join (
  file f32.zson
) on num=num otherword:=word

$ zq -I join-2.zed 
[no output]

This seems counterintuitive. As a user, I'd probably hope that comparison between all numeric types would succeed, but if not, I'd have expected comparison between float types to be the ones that would work first. The user came away from the experience with the impression that the join key expressions would now always need explicit casting on both sides, when in fact the int/float comparisons work fine as long as float64 is in use.

To me, this makes an argument for hurrying up and fixing these types or dropping them from the Zed tooling until we can invest the time to fix it right.

[philrz]

Verified in Zed commit 19b2eb5.

The looping runs above that perform simple math between all the numeric types now returns no output, which is a sign that it's running correctly (i.e., 5 - 4 = 1 in all cases).

$ zq -version
Version: v1.14.0-29-g19b2eb5d

$ for type1 in $(cat types.txt); do for type2 in $(cat types.txt); do   echo -n "$type1 $type2: " ; echo "{num1: 5($type1), num2: 4($type2)}" | zq -z "yield num1 - num2" -; done; done | egrep -v " 1$"\|" 1\.$"\|" 1\.\("\|" 1\("
[no output]

Also, the join-2.zed example above that previously produced no output because the equality was not working between the different float types now gives the expected output.

$ zq -I join-2.zed 
{num:1.,word:"hello",otherword:"goodbye"}

Thanks @mattnibs!