ROUNDING / ACCURACY ISSUES:
What is the value of 1/3? You might say .333333, but that would be incorrect. That series of 3's never ends, and if you stop showing the repeating 3s, then your answer is inaccurate.
The same thing happens with computers that work with binary numbers.
1/10 in binary is .0001100110011001100110011 - never ending. If you stop adding the extra digits, you get a number that is inaccurate.
All binary computers, from the beginning, have had to deal with this issue. Analyze is no different. If you are working with floating point numbers you are going to encounter "rounding issues", because the numbers are stored internally in binary form.
SOLUTIONS:
There are a couple of possible solutions when dealing with numbers that should be Currency:
- multiply by 100 and deal with the numbers as longs (convert to/work in Pennies)
- apply rounding as you go thru your computations, knowing that the final result may be slightly off
- apply rounding to the final calculation only
- use the python format() function to convert it to a string: '{:.2f}'.format(myNumber)
Note: even if you use the function round(1/10,2) you are still going to get a "rounding error" - the number is still .0001100110011.... and using round() simply returns the same value.
Note2: This "error" is new to many people because they've never studied base 2 arithmetic (binary), however, the rounding error has always existed. The trouble is, for systems that need highly accurate numbers (thousandths of a penny), Excel's extra rounding actually introduces errors.
For further information, this YouTube video explains it quite well:
Comments
1 comment
Also see the following community forum post regarding the use of Decimal type values in Analyze:
https://support.infogix.com/hc/en-us/community/posts/360035102494-Floating-point-division-returning-incorrect-values
Please sign in to leave a comment.