404 Error On Custom Taxonomy Pages 2, 3, etc

Question

I've been reading a bunch of posts about this with no luck so far. I can't seem to figure out what is the problem.

I have the following:

Custom Post Type: Sermons

Custom Taxonomy: Series

My urls right now look like this mysite.com/Sermons/ParentTerm/ChildTerm/Post

When viewing the Parent Term of the series taxonomy the first page works fine. However when navigating to the next page using wordpress' pagination it throws a 404. So trying to view mysite.com/Sermons/ParentTerm/page/2/ does not work.

I'm guessing I need to do a URL rewrite? I tried the following with no luck:

function sermon_rewrite_rules() {
  add_rewrite_rule('sermons/%series%/([^/]+)/?$',
  'index.php?post_type=sermons&%series%=$matches[1]', 'top');
  add_rewrite_rule('sermons/%series%/([^/]+)/page/([0-9]+)?$',
  'index.php?post_type=players&type=$matches[1]&paged=$matches[2]','top');
}
add_filter('init', 'sermon_rewrite_rules');

I've tried reading up on everything I can about this and it's all starting to blur together, any help would be greatly appreciated. Thanks.

EDIT

So after some tinkering around I have this working and allowing for paginaiton on taxonomy pages, but it's breaking my actual post now and displaying the terms there. I feel like I'm super close to the answer, any ideas?

function sermon_rewrite_rules() {
 add_rewrite_rule( 
    '^sermons/([^/]*)/([^/]*)/(\d*)?',
    'index.php?series=$matches[1]&p=$matches[2]&paged=$matches[3]',
    'top' 
 );
}
add_filter('init', 'sermon_rewrite_rules');