Okay, found the solution. Seems I didn't think it fully through.
The problem was the opacity being "inherited" by all colors of the view, while I only wanted the background to be opaque. So, I just made the background opaque :) Like so:
self.backgroundColor = [UIColor colorWithRed:1 green:1 blue:1 alpha:0.2];