Principal curvatures, if I am not missing a different definition, are defined on differentiable surfaces but meshes (usually a collection of triangles) are not differentiable. I can imagine some of possible approaches to approximate the principal curvatures.
Assuming that your mesh is obtained by sampling from a differentiable surface, what you need to do is polynomial regression, more specifically quadratic interpolation on the nearby vertices around the point you are trying to calculate curvature.
First you'll need to determine the normal vector at your point of interest. The normal vector can be obtained by the normal direction of a mesh triangle ( AB × AC , cross-producting two edges of a triangle), and interpolating with some other normal vectors.
Once you've found the normal vector, you can transform the coordinates of you mesh so that the point is at the origin and the normal vector is pointing the z axis. (with a parameter of rotating along z-axis by angle theta)
Then your next goal is to find a surface
1 1 z = --- alpha * x^2 + --- beta * y^2 2 2
that best approximates your set of points, with three parameters alpha, beta and theta.
Then alpha and beta is the principal components.
I am not certain but there will already be a matlab function that does this regression for you.