OK I worked this out. As documented here that API can't be used pre-layout.
The correct API to use is described as:
Note: Only use the simpler method newLatLngBounds(boundary, padding) to generate a CameraUpdate if it is going to be used to move the camera after the map has undergone layout. During layout, the API calculates the display boundaries of the map which are needed to correctly project the bounding box. In comparison, you can use the CameraUpdate returned by the more complex method newLatLngBounds(boundary, width, height, padding) at any time, even before the map has undergone layout, because the API calculates the display boundaries from the arguments that you pass.
To fix the problem I calculated my screen size and provided the width and height to
public static CameraUpdate newLatLngBounds(
LatLngBounds bounds, int width, int height, int padding)
This then allowed me to specify the bounding box pre-layout.