After researching it appears that the standard camera shutter sound is next to impossible to change via SDK.
This answer supports that. You can replace your own sound, but not in an app.
As for AVCapture, it also appears that you can't change the shutter sound as capturing images covertly is against the App Store policy. See this answer.
The only way to take a silent picture is to use a video screen grab, which you said you don't want to do for image quality reasons.
So AFAIK, there is no way to change the shutter sound not using video screen grabs.