It turns out Android will not release the activity from memory until the entire process is restarted. Since it is a foreground service, its process is almost never killed, so this doesn't happen.
The solution is to run the service and activity in their own processes by specifying:
android:process="name"
in AndroidManifest.xml