cuda-gdb can't see available devices

Question

I'm trying to use cuda-gdb debug my code on a remote cluster via ssh. There're 6 Tesla C2070 gpu on the node:

$ deviceQuery | grep "^Device"
Device 0: "Tesla C2070"
Device 1: "Tesla C2070"
Device 2: "Tesla C2070"
Device 3: "Tesla C2070"
Device 4: "Tesla C2070"
Device 5: "Tesla C2070"

But when I run code under cuda-gdb, it still gives me error code=24: "fatal: All CUDA devices are used for display and cannot be used while debugging. "

I've already set -gencode arch=compute_20,code=sm_20 and compiles well. When I type info cuda devices in the debugger, it spits No CUDA devices. I also tried export CUDA_VISIBLE_DEVICES="0,2" but still not working. What should I do? I'm using cuda 5.0. Thanks in advance.

Answer 1

Try running "lsof /dev/nvidia*", it will show you which program is holding the device nodes open. If X is using those GPUs, it will be evident from the output.