Coordinating Times From /sys/kernel/debug/tracing In Ubuntu

Question

I am trying to gather data from multiple sources on an Ubuntu 10.10 machine programatically about a program's performance. For all of my other sources, I have been able to gather them using the RDTSC x86 instruction, and then scale them using gettimeofday to convert to seconds in absolute time. When I start trying to coordinate these data sources with the output of doing sched_switch tracing in /sys/kernel/debug/tracing, however, I run into an issue, since the output I am seeing is in seconds and microseconds since some unknown time.

Steps I have already done:
1. I've determined that the Linux kernel internally also uses RDTSC, but adds some offset which it gathers, but I don't seem to have ability to retrieve. It does this on a per-core basis as well, which means I would have to try all four cores and determine the best one, which seems like a poor solution to this problem.
2. I've tried converting RDTSC times around turning on logging, to see if at least the conversion itself is consistent (i.e. some constant offset), but the scale doesn't seem to stay constant throughout the run.
3. clock_gettime(CLOCK_MONOTONIC, ...) seems to have a very similar value, but is always off by an unexceptable amount (around half a second), and doesn't seem entirely consistent either.

If I am able to change how my other data sources gather their time to whatever is needed (assuming it isn't performance intensive), how should I gather times in order to coordinate between trace's time and the times I am gathering? Is there some sort of way to change the output to RDTSC, so I can just use that, or is there a system call I can make to get the same time as what is being output to trace? Thanks in advance for any help.

Answer 1

So after looking at the source code, I found that at one point, the code takes the RDTSC value, and does some fancy math with it to try to multiply it by the clock frequency and add some offset that is calculated on boot up. This code seems to be a bit outdated, however, since RDTSC is guaranteed to be consistent across cores on newer chips, while this assumes each one is different. It also seemed to be gathering the current frequency instead of the max frequency, which is what the documentation seems to state that RDTSC uses.

Thus, at this point it seems that the time isn't directly relateable to actual time at an accuracy level that is useful in my instance. Hopefully this bug is fixed in an upcoming kernel release to fix this, and open up the opportunity for me to synchronize these two sets, but until then, it doesn't seem reliable enough.

Answer 2

RDTSC is not a popular instruction. There are some problems when using it, some events like hybernation resets the counters. Other common source of problems is dynamic CPU clock.There is good post comparing rdtsc and hpet:

http://aufather.wordpress.com/2010/09/08/high-performance-time-measuremen-in-linux/

To check rdtsc accuracy, I measure how many clock cycles it counts in one second and compares it to the declared cpu clock. It only works with CPU dynamic clock disabled. See:

https://github.com/petersenna/rdtscbench