9 Replies Latest reply on Jun 27, 2016 5:04 AM by Mark Fraser

# Calculation showing as false

Hi All,

I am writing a simple calculation i.e. 1.2/0.05 it is giving answer as "24" which is correct. When we are using this to check a condition. (See attached screen shot) it is giving answer as "False". Please find screenshot. Anybody have idea about this behavior.

Thanks

• ###### 1. Re: Calculation showing as false

Interesting...

I'm afraid I don't know, I'm replying to this thread as I'm interested if anyone knows why

Testing...

1=1 > TRUE

1=2 > FALSE

1*1=1 > TRUE

This evaluates to 24

1.2/0.05

All as expected. Then...

IF 1.2/0.05 >= 23.99 and 1.2/0.05 < 24 THEN 'true' ELSE 'fail' END > TRUE???

1 of 1 people found this helpful
• ###### 2. Re: Calculation showing as false

it doesn't seem to be a problem because they are fractions

• ###### 3. Re: Calculation showing as false

Thanks for the response MARK FRASER!

As per the arithmetic calculations 1.2/0.05 = 24. Even in tableau it is showing as 24.000000.

Thanks

• ###### 4. Re: Calculation showing as false

I thought it was me, but I don't 'get it'.

I have demonstrated above that Tableau is usually OK with calculations (in whatever form) but this is just weird.

For (reasons I don't understand) its evaluating the calculation as somewhere between 23.999999999 and (less than) 24, but not 24!?

Simon Runc, Shawn Wallwork, Tracy Rodgers, Yuriy Fal any ideas??

I'm using 9.3.3

• ###### 5. Re: Calculation showing as false

I think this is part of the whole rounding issues that have been lurking around for awhile. Here's more weird:

And:

--Shawn

• ###### 6. Re: Calculation showing as false

Yep so here's the 'workaround':

1 of 1 people found this helpful
• ###### 7. Re: Calculation showing as false

Hi all,

Follow Shawn in suspecting a "rounding" problem.

Please find the attached (with parameters).

Yours,

Yuri

PS  Interesting reading on a subject:

The price of correctness

1 of 1 people found this helpful
• ###### 8. Re: Calculation showing as false

1.0 is rarely equal to 1 and most of software engineer been bitten by this once or twice.

It all comes down to differences in how decimal numbers are stored vs how whole numbers are stored.

Here's fun read: Double-precision floating-point format - Wikipedia

Or this (for normal people): Why is 0.1+0.2 not equal to 0.3 in most programming languages? - Quora

**** computers!

When comparing decimals (especially results of arithmetical operations or aggregations) it's better to avoid equality comparisons. I usually do [N1] - [N2] < 0.00000001 or something like that.

2 of 2 people found this helpful
• ###### 9. Re: Calculation showing as false

Thanks all, very interesting discussion

sharath kumar

I hope the above is useful, I have certainly learnt quite a lot!

Assuming you have what you need - please can you mark 1 or more answers correct/ helpful? (i.e. Dmitry's, Yuriy's and Shawn's, thank you!)

Thanks again all, a fantastic advert for this community!!

Cheers

Mark