Fixed by adding two gestures. One called "left" and one called "right" and they're shaped like horizontal lines. When predicting the closest gestures, it counts as a failure if left or right have the highest scores.
This prevents users from swiping left or right to draw a complex shape.