What's the best way to do this? a reference from the IIS server to the database server using a connection string? won't it slow down performance?
There is no best way because there is only one way to start with. Put the SQL Server on another machine. Finished.
You will have to adjust the connection string. This will introduce latency and the bandwidth is lower than in memory, but practically those things are irrelevant - unless you make a select * From table
and filter in memory, which hopefully no one does.
Point is - you get more processing power, you get scalability, and the small theoretical impact is neglegible in comparison. If you manage to load up even a 1gigabit network link with data you have either some seriously bad design or some really awkward programming. The higher speed of a non trivial and hopefully build for the purpose database server will far more than outweigh eny small loss of performance.
Does replicating the database to this third services server a good idea?
How long is a piece of string? It depends - on the services. In general - no. In very few edge cases - yes. How to decide? Experience - and common sense, based on the specific requirements.