1 of 1 people found this helpful
I'm afraid I don't know, I'm replying to this thread as I'm interested if anyone knows why
1=1 > TRUE
1=2 > FALSE
1*1=1 > TRUE
This evaluates to 24
All as expected. Then...
IF 1.2/0.05 >= 23.99 and 1.2/0.05 < 24 THEN 'true' ELSE 'fail' END > TRUE???
I thought it was me, but I don't 'get it'.
I have demonstrated above that Tableau is usually OK with calculations (in whatever form) but this is just weird.
For (reasons I don't understand) its evaluating the calculation as somewhere between 23.999999999 and (less than) 24, but not 24!?
I'm using 9.3.3
2 of 2 people found this helpful
1.0 is rarely equal to 1 and most of software engineer been bitten by this once or twice.
It all comes down to differences in how decimal numbers are stored vs how whole numbers are stored.
Here's fun read: Double-precision floating-point format - Wikipedia
Or this (for normal people): Why is 0.1+0.2 not equal to 0.3 in most programming languages? - Quora
When comparing decimals (especially results of arithmetical operations or aggregations) it's better to avoid equality comparisons. I usually do [N1] - [N2] < 0.00000001 or something like that.
Thanks all, very interesting discussion
I hope the above is useful, I have certainly learnt quite a lot!
Assuming you have what you need - please can you mark 1 or more answers correct/ helpful? (i.e. Dmitry's, Yuriy's and Shawn's, thank you!)
Thanks again all, a fantastic advert for this community!!