You say "Just the record that was inserted". Inserted
can contain more than one record. If there is only one, then your trigger will function as you expect. But if there is more than one, it won't.
I would rewrite your logic into a single update
statement along the lines of...
Update CustomerContact
Set Deleted = 1
From CustomerContact
inner join inserted on CustomerContact.CustomerContactID = inserted.CustomerContactID
inner join orders on inserted.OrderID = orders.OrderID
where
-- some criteria.