You would think that maybe there is a named part in the control template
Yes ... it turns out there is an "undocumented feature". Trial and error shows that the "BoxMiddleLine" is the named part that controls the selection. So adding to the control template a clickable element with that name, fixes the issue:
<Grid>
<Border x:Name="BoxMiddleLine" HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Background="#0000" />
<ToggleButton x:Name="toggle"
IsChecked="{TemplateBinding IsChecked}"
IsHitTestVisible="False">
<ContentPresenter x:Name="contentPresenter" />
</ToggleButton>
</Grid>