The key is associated with a user account, meaning one needs credentials to access the key. If a hacker can hijack that identity, your concerns apply. However, simply copying or stealing the IO subsystem is insufficient to break the encryption, nor is accessing the IO subsystem with a different account.
Additionally, for complete security, backup media have to be protected as well. Backing up unencrypted data creates another attack vector. If you run a SaaS for larger clients, or if you must comply with standards such as HIPAA or PCI, you may be required to ensure backups area encrypted.