Unlike for A32 ("ARM") and T32 ("Thumb") within a 32-bit toolchain, there is no cross-instruction-set support between 32-bit and 64-bit ARM architectures. A64 really is a completely new instruction set.
Since interworking does not exist between the 64-bit and 32-bit states (you can only change 'width' on taking an exception) this is not an issue in normal usage. But when building a standalone image containing code for several exception levels I can imagine it will get a bit tedious.
You need to use aarch64-linux-gnu- for A64 portions and arm-linux-gnueabihf- for A32/T32 portions.