To answer your question, the ramifications of removing arm64 from the architectures is that some classes of application will perform slower on devices with 64-bit capable processors (right now just iPhone 5s) than they would with arm64 builds enabled.
Note: at this point arm64 is required for all applications.