FTP does not work interchangeably with HTTP, they are separate servers/protocols. I'm assuming you added "anonymous access" to your HTTP server from your description. Just installing the FTP service is not enough. Did you set it up?
Also your Host for the FTP server will always be just the IP (or domain.com) address. "Connection refused by the server" means the server isn't running or you're trying to connect to an invalid port (default port for FTP is 21).
Did you access the FTP server console and set up a server pointing towards the directory you want to serve, credentials, etc.? This is not the same thing as the WebSite manager.
For the moment pretend your HTTP server doesn't exist and just focus on FTP.
This should look familiar: