Here you have a detailed discussion about the jitter and RTP.
The main aspect you are interested is the following:
In the Real Time Protocol, jitter is measured in timestamp units. For example, if you transmit audio sampled at the usual 8000 Hertz, the unit is 1/8000 of a second.
In that page they also mention the relation between the timestamp and the receiver: The difference of relative transit times for the two packets is computed as:
D(i,j) = (Rj - Ri) - (Sj - Si) = (Rj - Sj) - (Ri - Si)
Si is the timestamp from the packet i and Ri is the time of arrival for packet i. You can also check to some examples there.