I personally haven't ran into any issues with mixing the code first fluent api and data annotations. I also wondered if there would be any crossover pain and I can honestly say I have yet to find any. Here's a few references to case studies on the subject to ease your mind.
(Direct from the EF team) http://msdn.microsoft.com/en-us/data/jj591583.aspx
(Part 1) http://www.codeproject.com/Articles/476966/FluentplusAPIplusvsplusDataplusAnnotations-plusWor