You should probably be doing this with a private network: http://docs.vagrantup.com/v2/networking/private_network.html
Once you have added the IP address into your Vagrantfile like this:
config.vm.network "private_network", ip: "192.168.50.4"
You should reload vagrant and be able to browse to 192.168.50.4, there should now be no problems because you will not need to use a port in the URL.