Question

Why exactly is Octave giving me the following results when given 1/3 or 0.1?

octave:1> (4/3-1)*3-1
ans = -2.2204e-16

octave:2> (10.1-10)*10-1
ans = -3.5527e-15

octave:3> (10.125-10)*8-1
ans = 0
Was it helpful?

Solution

This is a consequence of floating point arithmetic. Your computer cannot fully accurately represent 4/3 as a floating point binary number hence you get a rounding error. However it can fully accurately represent 10.125 in binary and hence no error.

Said another way, consider the parts of your number after the decimal point. For 4/3 we have 0.333333... which is 0.010101010101... in binary. Your computer can't store an infinite series of numbers after and so it truncates to some precision. However for 10.125 consider what 0.125 is in binary which is simply 0.001. So only three "decimal" points in binary and thus no truncation and no rounding error.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top