I would use a regular expression validator as follows on your model:
[RegularExpression(@"#([0-9][A-F])*", ErrorMessage = "Invalid Color Code")]
This will include the "#" in the validation, so it will pass validation, then you can deal with the "#" on the server side, if you need to.
If you can't change the model validation, maybe you could do something in JavaScript when the before the form is submitted to remove the "#" from the value that is submitted.