As Guntram Blohm pointed out:
Maybe the access import module reads the BOM, decides on UTF-16, and has a fallback to UTF-8 if the next 2 bytes are not a valid codepoint. This seems to be the case, it may be a bug, it may be that UTF-8 with BOM was not implemented completely, I don't know.. If you do feel free to drop a comment, I'm still curious.
Anyway, there are 3 ways you can "solve" the issue:
Remove the BOM (rewriting the entire file, could be done in memory rather than reading & writing from disk) see here: http://axlr8r.blogspot.nl/2011/05/how-to-export-data-into-utf-8-without.html
Remove the double quote or place an extra character after the BOM if you can.
Save the file as UTF-16 which Access seems to have better support for. Compared to an ANSI file a UTF-16 file will be twice the size. Whereas UTF-8 would only add extra bytes per character when special characters are met. In my case the file is in an intermediary state, I'm not keeping it, after importing I delete the file. So I chose to save the file as UTF-16 and import it with Codepage 1200 (which is UTF-16 for the rest of the world).