I figured out the right approach. The correct way to do this is to find and modify the ServiceBehaviorAttribute object that is attached to the service description:
var transactionTimeout = TimeSpan.FromSeconds(...);
var behavior = serviceHost.Description.Behaviors.Find<ServiceBehaviorAttribute>();
behavior.TransactionTimeout = transactionTimeout.ToString();
serviceHost.Open();