Actually, calling setMultiChoiceItems does not result in CheckBox widgets but CheckedTextView widgets. [Updated] It also seems that changing the value for the checkMark attribute does not have any effect. However, by changing the value of the listChoiceIndicatorMultiple attribute for the CustomDialog style, I was able to change the drawable for the choices. Like this:
<style name="CustomDialog" parent="@android:style/Theme.Holo.Light.Dialog">
<item name="android:listChoiceIndicatorMultiple">@drawable/btn_check</item>
<item name="android:background">#aaf</item>
</style>
I discovered this from looking at the Android source code and finding that the template used for the multiple choice items in the dialog box is this one: