In MATLAB R2013a, LU, QR and SVD are all supported on the gpu through gpuArray. There's a list of supported functions here: http://www.mathworks.co.uk/help/distcomp/using-gpuarray.html#bsloua3-1 . The linear algebra functions of gpuArray are all implemented using MAGMA.
A\b
can run efficiently on the GPU, see this example: http://www.mathworks.co.uk/help/distcomp/examples/benchmarking-a-b-on-the-gpu.html . As the problem size increases, you can expect a decently powerful GPU (e.g. Tesla) to run up to 5 times faster than the CPU.