Have same issue when Microsoft Irina Desktop voice is available in system. Workaround to set voice explicitly in prompt, e.g.:
using System.Speech.Synthesis;
var synth=new SpeechSynthesizer();
var builder=new PromptBuilder();
builder.StartVoice("Microsoft David Desktop");
builder.AppendText("Hello, World!");
builder.EndVoice();
synth.SpeakAsync(new Prompt(builder));
As you already utilize PromptBuilder, try to add StartVoice and EndVoice calls around text.