After looking at this further, I have decided that the simplest method here is to just change the icon images.
Instead of having key icons consisting of a number/background, The icons will just be numbers with transparent backgrounds and a separate, number-less background drawable will be used in place of the KeyboardView
keyBackground
.