The key to making this work without any lag / delay in the audio or video is to store the view with the AVPlayerLayer outside of the view. When reloading the controller, instead of creating a new view and assigning its player to the same player, simply attach the old view to the new view controller's view.
The view stays in memory as long as the video is still playing, that way a new AVPlayerLayer is not created and assigned to. It is the reassigning that causes the lag.