Found out how to do this after a couple days of monkeying around. I added the authn_file module and then also added AuthBasicAuthoritative off. This seems to still force authentication with mod_auth_apple but not fail when the AuthUserFile isn't specififed. It also passes the authentication information on to authz_svn so it can check acess control properly. I'm hoping someone who's more familiar with apache can properly explain why this works.
So my location directive now looks like
<Location "/">
Options All -Includes -ExecCGI -Indexes +MultiViews
<IfModule mod_dav.c>
DAV svn
SVNPath /Users/path/to/repo
</IfModule>
AllowOverride None
AuthName "Svn Repo"
AuthType Basic
AuthzSVNAccessFile /Users/path/to/access/file.txt
AuthBasicAuthoritative Off
Require valid-user
</Location>
Annoyingly, I do have to add AuthBasicAuthoritative Off to any other sites my server hosts.