There are two ways you can accomplish this. The first is to use a Style targeting the CellValuePresenter where you change the Template. If you are not editing the values, this is the recommended approach and there is an example in the Embedding a Custom Control in a Field help topic.
The second way to do this is to create a custom ValueEditor which is better if you need to allow editing. You may be able to use the ControlHostEditor that Andrew Smith provides in the sample in his Hosting WPF Controls in a XamDataGrid Cell blog post.