Question

Have you ever observed this phenomenon:

If I try to calculate the following calculation in ImmediateWindow of Visual Studio 2010 C# .Net, I get this weird solution.

1E-9 * 100 = 0.00000010000000000000001
1E-9 * 1E+2 = 0.00000010000000000000001

Even if I do the calculation this way:

double val = 1E-9;
0.000000001
val = val * 100;
0.00000010000000000000001

If I calculate the same calculation in Windows calc I get the expected result:

1E-9 * 100 = 0,0000001

If someone knows something about this behaviour I would be very pleased to get the info.

Was it helpful?

Solution

It doesn't matter if you use exponential notation or not, you're just running into floating-point accuracy. If you want better accuracy, use decimal e.g.:

decimal val = 1E-9;
0.000000001
val = val * 100;
0.0000001

See What Every Computer Scientist Should Know About Floating-Point Arithmetic

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top