TEASER icon indicating copy to clipboard operation
TEASER copied to clipboard

Number of decimals in records

Open marcusfuchs opened this issue 4 years ago • 4 comments

Currently, TEASER outputs records with a very high number of decimals, e.g.

...
    UWin= 1.1034256322858929,
    ratioWinConRad = 0.02,
    AExt = {148.12145999999998, 148.12145999999998, 148.12145999999998, 148.12145999999998, 159.44039999999998, 159.44039999999998},
    hConExt = 2.350107008525131,
    nExt = 1,
    RExt = {4.262502303317977e-05},
    RExtRem = 0.00454563643967992 ,
    CExt = {255007830.43196833},
...

We see that depending on different platforms (Linux, Windows), some of these values can change in very minor decimals. Is there any objection to limiting the number of decimals to a meaningful limit to avoid this and also to make the records easier to read? I would expect that we cannot meaningfully calculate e.g. the value for RExt to a precision of 1e-20 as is done currently.

marcusfuchs avatar Jan 04 '21 09:01 marcusfuchs

@MichaMans We are again experiencing some problems related to this issue. As there was a long silence, I think there are no objections and we could implement a fix?

marcusfuchs avatar Mar 09 '21 15:03 marcusfuchs

@marcusfuchs no objections, go ahead 😃 And thanks in advance!

MichaMans avatar Mar 09 '21 17:03 MichaMans

I would appreciate this fix as well. Have you made any progress on this @marcusfuchs? How many decimals are meaningful to you? I think a limit of 3 would be plenty for my purposes, and expect I'll be happy with any improvement over the current behavior.

vtnate avatar May 20 '22 18:05 vtnate

@vtnate I'm really sorry, but unfortunately my priorities had shifted right after my last comment and I'm not too optimistic that I'll get to propose a fix in the near future. But yes, I was also thinking in the same direction of about 3 decimals.

marcusfuchs avatar May 23 '22 11:05 marcusfuchs