Is it possible for you to merge the two "main layers" of your UIScrollView into one? Also, is it possible for you to add layers directly to UIScrollView's layer instead of adding additional ones?
I find that I get huge performance wins by reducing the number of layers that exist for the sole purpose of containing other layers.