If I buy an NVIDIA GPU with PCI express enabled, can I connect it to my system and run CUDA?
Yes, sure! Just make sure that you have the correct PCI express slot and a correct Power Supply Unit (some high-end GPUs need an extra power line, because PCI express can provide a limited amount of current).
How much bottleneck comes in the memory transfer between host and device(CPU-GPU)?
Theoretically GPUs sitting on PCIe 2.0 x16 are capable of 16*5GT/s, that because of the error correction encoding (2 ecc bits every 10 bits, 8/10 encoding) become 16*4Gbit/s: 8GB/s.
If your systems has x16 PCIe 3.0 as the same as the GPU, the theoretical transfer rate is: 16*8GT/s. Here the error correction scheme is much more lightweight (128/130 encoding), so it's approximatively 16*8 Gbit/s = 16GB/s.
I've seen for real up to 6GB/s on PCIe 2.0 systems - never touched one with PCIe 3.0 actually.