It depends if you used the Button class for your keys. If so, there is no way to define what you call tolerance.
To avoid that, I'd create a class Key extending view or TextView. It would be less simple because you would have to define a background yourself, a font size; but it will be way more flexible, because you could create a custom hitbox a bit smaller than the button (and defined by a Rect). For instance, your button size less a margin of 5dp.
Then in the onTouchEvent, i'd check if the finger position is inside the hitbox. If it's not, simply return to the caller.