Question

Pretty strange things happen here:

float theFloat = 10f * 0.5f * 0.0502913967f;

//result is 0.2514569835f
//but a float's precision is only 7digits so it's false.    
//I corrected it, rounded at 6digits, result is 0.251457f

float theFloat = (float)Math.Round((decimal)(10f * 0.5f * 0.0502913967f), 6);

That's okay, but when I use it with another float:

Vector2 theVector2 = new Vector2(16302.51f, 0f);
theVector2.X += theFloat;

//means : (16302.51 + 0.251457) = 16302.761457
//but when I check the var the result is : 16302.7607
//so I don't get it...
//also, if theVector2 is 200f more than the previous example (=16502.51f, 0f),
//result is 16502.7617 (200.001f more than previous example's result)

Where is my error? Hope you can help.

Was it helpful?

Solution 2

The answer is : Store the "number with too many digits" in a double, Round it and cast it in a float.

OTHER TIPS

There's no real error other than your understanding of the different variable storage types and their precision. I'm just answering your question, not trying to be rude.

Also, you are using an int, float, and double all in one equation, and then casting all of that to a decimal. This is very bad practice, as all of them have different precision. In general, use the smallest storage type necessary, and use the same types unless absolutely necessary, which requires cast (implicitly or explicitly) and changes precision.

BlackBear's link is very good.

Also see these SO answers:
https://stackoverflow.com/a/618542/1002098
https://stackoverflow.com/a/10056298/1002098

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top