They're completely different things:
- GLSL is the language used to write shader programs that run on the GPU. It's a variant of C with some special OpenGL-specific extensions. But as far as your application is concerned, a GLSL shader is just an opaque data file to be passed to the OpenGL library; it's completely independent of the host program.
- GLM is a C++ library for working with vector data on the CPU. For convenience, it follows similar naming conventions to GLSL, but it's completely independent of OpenGL.
GLM isn't meant as a substitute or alternative to GLSL. It's meant to help with calculations that wouldn't make sense to do on the GPU — things like building a projection matrix to be used by your vertex shaders, or computing distances between points in 3D space.