Using a rewrite map is fine. There's a reverse way to do this that I've posted about here.
You can simplify the rules a little though:
RewriteMap ipslist txt:/path/to/whitelist.txt
RewriteCond ${ipslist:%{REMOTE_ADDR}|black} ^black$ [NC]
RewriteRule ^ - [F]
The whitelist.txt file needs to look like:
1.2.3.4 ok
2.3.4.5 ok
etc.
The "ok" can be anything, but you need something that the whitelisted IP address maps to, other than "black". The whitelist.txt file will be cached by apache and when you change it, apache will automatically reload and reparse the file. This way, you don't need to restart apache.