5 Replies Latest reply on Nov 21, 2016 12:24 PM by J Shap

Tableau is not performing a calculation correctly

J Shap Nov 21, 2016 9:18 AM

I am doing a simple calculation in tableau. all I am doing it dividing one metric by another. tableau is creating some strange rounding error.

for example when I divide 3 by 5 I should get .6 but Tableau is giving me 0.600000023841857.

I am using an extract using a SAS data set.

is this a know issue when using extracts created with a SAS dataset?

when I build an extract from an excel spread sheet I do not get the same rounding errors.

Thanks

Joe

1. Re: Tableau is not performing a calculation correctly

Matthew Risley Nov 21, 2016 9:22 AM (in response to J Shap)

Hey J Shap,

In the SAS dataset, using your example are the numbers exactly 3 and 5? Tableau will pull in whatever the data is.

If this is causing you to report the wrong numbers you can use the ROUND () Function to "solve" this.

Best,
--Matt
Like Show 0 Likes(0)

Actions
2. Re: Tableau is not performing a calculation correctly

Rekha Kesavan Nov 21, 2016 10:06 AM (in response to J Shap)

Hi Shap,

If you are specific about the decimal points, please change the Default properties of the Calculated filed to Custom Number--> Decimal--> .2 places.
Like Show 0 Likes(0)

Actions
3. Re: Tableau is not performing a calculation correctly

J Shap Nov 21, 2016 10:29 AM (in response to Rekha Kesavan)
thanks guys for your reply. I am attaching a sample workbook.

you can see that the variables I am using are whole numbers.

I am using the percent's to create percentage buckets so just formatting the percent variable doesn't get me my desired result.

I just cannot figure out why the calculated variable is not performing the division correctly.

if you see anything please let me know.

Thanks

J

Book4.twbx 54.3 KB
Like Show 0 Likes(0)

Actions
4. Re: Tableau is not performing a calculation correctly

Matt Hong Nov 21, 2016 11:25 AM (in response to J Shap)

There are always annoying errors involving floating point numbers. This has nothing to do with Tableau, it's just the nature of computing systems.

It is also why you should always set a measure as a whole number if it is an integer, or a float if it contains numbers with decimals. The solution is to set both 'Total Spanish Language' and 'Units' as whole numbers:

Right click on measure -> Change Data Type -> Number (whole)
Like Show 0 Likes(0)

Actions
5. Re: Tableau is not performing a calculation correctly

J Shap Nov 21, 2016 12:24 PM (in response to Matt Hong)

Thanks soooooo Much! such a simple solution.
Like Show 0 Likes(0)

Actions