why are floating point operations considered expensive?

Question

I read that gprof (function profiling) and other methods of profiling can return the number of floating point operations taking place in the execution of a program and thus was wondering how Flops are so much more expensive than regular operations?

Answer 1

I'm going to assume you're talking about x86, but a lot of the below applies equally well to other architectures

Floating point operations are expensive because operations on floating point numbers are much more expensive than operations on integers. It's that simple. The format of integers makes addition and subtraction extremely simple to implement in hardware. Floating point numbers are (almost always) implemented in IEEE 754, which stores numbers as a sign, exponent, and mantissa, which allows for the representation of very large and very small numbers, but it comes at the cost of operation speed. If numbers only had 3 decimal places, you could use integers, and just divide by 3 at the end; the wide range of precision complicates things.

That being said, modern processors are better with floating point numbers than they used to be. Floating-point math was originally implemented on an optional coprocessor - the Intel 80387 in particular - and it could only be accessed with special instructions. You'd push values onto the x87 stack, perform an operation, and then pop it back into a hardware register. Very slow, because it had to leave the processor. Even more importantly, those particular operations became "risky" to use because you couldn't be sure the processor existed - if it didn't, your program would work, but it would use software routines that emulated the coprocessor. If you were a game developer and you couldn't rely on, say, inverse square root being fast, you could do the work yourself, and you'd run equally fast on all systems - not slightly faster on some, and much slower on others.

Nowadays, processors have special floating point operations that are designed for performance, and more importantly, guaranteed to be there. So they're very fast, and while floating point ops are unavoidably slower than integer ops, they're not usually enough of a problem to do anything about it - especially at the expense of bugs and complexity. What's more, this answer suggests that, in most cases, it's a wash.

In any case, performance is now good enough that the old adage kicks in - programmer time is more important than machine time, and you'll definitely spend a lot more time on programming some fancy algorithm avoiding floating point numbers than you would by just using them.