If modified since with (replace www) modrewrite-rule

Question

i only use the my domain http://myDomain.com and not http://www.myDomain.com,
so i use this rule

RewriteCond %{HTTP_HOST} ^www\.(.*)$
RewriteRule ^(.*)$ http://%1/$1 [L,R=301]

my problem is, that i always get an 301 redirect -> even when i visit my domain without www

Any ideas to fix this?

My goal is:

http://www.myDomain.com -> 301
http://myDomain.com -> 304 Not Modified

Thanks in advance!