Sounds like you are running a debug build from withing VS, never the best way to measure the real performance as that means runtime optimizations are disabled.
Anyway if you are looking for the maximum performance WF4 is not the thing. Due to the asynchronous nature of WF4 execution it is always going to be somewhat hard to predict. And there is always ging to be a bit of overhead with the runtime and all activities involved which means that a simple C# function is always going to run faster.