If you want the server to not have the data, you have to make it such that the server never gets the data. The client needs to do the encryption, using key material which only exists on that client. (e.g. if the key is sent to the server then this provides no protection)
If you want multiple devices to be able to access the data, then you need to base the key you use for encryption on something the customer knows and can tell the device, such as a password or pass code. Basing it on device identifiers or something like that obviously won't work if you want multiple devices to be able to access the data.
Cryptography is "mechanical leverage" -- it lets you protect a large secret (the cleartext) using a small secret (the key).
As for performance, I'd be HUGELY surprised if encryption and decryption made more than a couple of percent difference in server performance, especially on platforms supporting AES-NI.
(Of course, if the attacker already has the data, they have the data. There's nothing you can do after the fact)