Typically you will create a xunit class called CommandRestClientTest to test CommandRestClient. You can hardcode a ServerAddress constant in that class to hardcode the serverAddress and pass it to every CommandRestClient instance you create in that xunit class.
Bear in mind that if you are actually testing a send method to somewhere that is an integration test. To be a pure unit test you would mock external interaction and only test business logic in CommandRestClient
In Unit you will typically put this in the initialization marked as [Setup] but xunit encourages you to create the object in every method.
sample code(haven't compiled it)
public class CommandRestClientTest
{
const string testServerAddress = "localhost:8080";
[Fact]
public void TestSomeMethod()
{
CommandRestClient commandRestClient = new CommandRestClient(testServerAddress);
//test, assert etc
}
}