setup Cloudfront to never refetch assets

Question

I have a Rails app that uses Cloudfront as its CDN and sprockets to compress assets and set long Cache-Control headers.

First request to an assets gives me response headers like so:

Cache-Control:public, max-age=31536000
Connection:close
Content-Encoding:gzip
Content-Type:application/javascript
Etag:"892a3afb514e3c84646835f9ef101026"
Last-Modified:Tue, 17 Jan 2012 16:05:17 GMT
Server:thin 1.2.11 codename Bat-Shit Crazy
Vary:Accept-Encoding
Via:1.0 c662f4e5a3bc7b224ce1bbecb0a23d82.cloudfront.net:11180 (CloudFront), 1.0 fc4797024fafe16fcc03e892c23f078d.cloudfront.net:11180 (CloudFront)
X-Amz-Cf-Id:bd2a89fb81bba2513f30d6940347693ef483defb4783da06d81b002af23545c39b9176b2c7cb144f,VF69FYlYU8ezzHQp7QkU2GvEQ8NmDqEwRX-wu7H63yNqE1ZlTaAq3g==
X-Cache:Miss from cloudfront
X-Runtime:0.012413
X-Ua-Compatible:IE=Edge,chrome=1

Perfect. It missed the cache from Cloudfront, fetched it from my app, so it should load it from there now from now on.

Now... if I refresh the page I get these response headers:

Connection:keep-alive
Server:thin 1.2.11 codename Bat-Shit Crazy
Via:1.0 86123d99569c9296c8605243e9a10621.cloudfront.net:11180 (CloudFront), 1.0 de9fc23d505dae3d19973a98bfa1eba0.cloudfront.net:11180 (CloudFront)
X-Amz-Cf-Id:a7bf6ffb30c43d306835190d210aeb8ec794c1ff619ad7aaecbad220c1822fe165bda624ae8382e1,bcij_88KOAvLPSqEKDRSvOBjEj7atiKBCAjSbqc2cfdb-BHOTi--LA==
X-Cache:RefreshHit from cloudfront
X-Runtime:0.006909
X-Ua-Compatible:IE=Edge,chrome=1

Looks like it hit Cloudfront. I get a 304 not modified. So it would seem that I'm not downloading the file again HOWEVER, I see this request also being made to my source server.

Why does this happen? Shouldn't Cloudfront always cache my assets now? I'm versioning my assets, so I really NEVER want this to be fetched from the server save for the first request. Is there any way I can force this type of behaviour?