Going to answer my own question on this one, oh well.
To fix this, I used this for my code:
<ComboBox
DisplayMemberPath="CustomerId"
SelectedValuePath="CustomerId"
SelectedValue="{Binding Path=CustomerId, Mode=OneWay, UpdateSourceTrigger=Explicit}"
ItemsSource="{Binding Mode=OneWay}"
IsSynchronizedWithCurrentItem="True"/>
If I don't specify the binding in the selected value to be a one way binding, it will then update the source. I thought the oneway binding in the itemsource was supposed to do that for me, but apparently that's not the case.