Operation is a specification-only element - imagine it as the method signature in OO programming languages. It has a name and a list of parameters.
Behavior is (among other things) what an operation (or another behavioral feature such as a reception) does when invoked - imagine it as the body of the method.
UML actually calls "method" the behavior that defines what an operation does. Also, from a behavior (be it an activity or a state machine), the operation is seeing as the "specification".
Note also that in UML operations can have multiple methods. What it means and what behavior(s) should be executed when an operation is invoked is up to the tool in question.
Finally, behaviors can be state machines or activities - activities are easy to understand, as they are the equivalent of procedural code. State machines are a totally different beast, and I admit I don't understand how a state machine can be used as the behavior for an operation.