The answer to your question:
- it possible for me to just keep the map bounds from changing after the markers are added?
is yes. And as you say, involves removing the map.fitBounds.
To get the map to center and zoom on a single address, either use the geocoder for that (implement it separately) or check the number of results, if it is one, center the map on the result and if there is a suggested viewport in the result, use that to center and zoom the map [with map.fitBounds(place.geometry.viewport) I think].