Short answer: No.
Generally speaking, the referrer - like all other HTTP headers - is user submitted data. User submitted data should not be trusted.
Off the top of my head I can think of these scenarios, where the referrer wouldn't be set without the user having manually entered the URL.
- Clicking a link in an email client.
- Using privacy software that removes referrers.
- Bots/spiders might not set this header.
- Following a link from a HTTPS site to a HTTP site.
There might well be other cases (fx I am not sure how browsers handle referrers when opening links in new windows).
And then there are the cases where the referrer is set, but no user has entered the address; notably bots - especially the referrer-spamming ones.