You are reversing the value. Stop reversing it and set the value to false.
ng-click="alert.latitude = false; alert.longitude = false;"
UPDATE:
ng-click="displayMap = !displayMap; alert.latitude = false; alert.longitude = false;"
But as this is getting a bit verbose I would probably create a function in my controller to reset these values. As far as I know with ng-click you can make it do as many things as you want as long as they are semi-colon separated.