I suppose you have the kernel source tree (from your distro package manager) on your arm device and you don't cross-compile :
make menuconfig
Navigate to : Device Drivers -> USB Support -> USB Serial Converter Support
And choose 'M'odule for USB FTDI Single Port Serial Driver
Exit, save changes and
make M=drivers/usb/serial/
make modules_install