There is not enough information here to completely answer your question - please add the version of MPSS and the current clocksource settings; however,...
Please have a look at dmesg and the current clocksource to make sure you're using TSC and not experiencing some problem. Also, read this excellent article from Ravi Murty to help you understand some of the problems that you might be experiencing with the Xeon Phi clocksource:
I'd propose that your current clocksource is set to micetc, which is incurring additional overhead from reading out of the mmio space of the device everytime your code (or the kernel on your behalf) wants to read the time. Switch to TSC to avoid that. Newer versions of MPSS should be setup with TSC as the default clocksource, but please read the article from Ravi and make sure your device is setup properly.