Pointwise mutual information vs. Mutual information?

Question

I am learning about information theory and mutual information. However, I am quite confused with MI(Mutual information) vs. PMI(Pointwise mutual information) especially signs of MI and PMI values. Here are my questions.

• Is MI values a non-negative value or it can be either positive or negative? If it is always a non-negative value, why is it ?

• As I search online, the PMI can be positive or negative values and the MI is the expected value of all possible PMI. However, expected value can be positive or negative. If MI is really the expected value of PMI, why is it always positive ?

Did I misunderstand anything of MI and PMI here ? Thank you very much,

Answer 1

PMI can clearly be negative, since if $p(x,y)<p(x)p(y)$, then taking $\log \frac{p(x,y)}{p(x)p(y)}$ gives a negative number.

But MI is always non-negative, as it can be written as $d_{KL}(p(x,y)||p(x),p(y))$, where $d_{KL}$ is the Kullback–Leibler divergence which is non-negative (the proof of the latter is not completely trivial).

Intuitively, this happens because if there is a point such that $p(x,y)<p(x)p(y)$, then the probability must "compensate" for this later, with a pair $p(x',y')>p(x')p(y')$.