There is no right or wrong here. Following REST and RFCs too closely may prove to be difficult when your API development is driven by specific client requirements.
In reality, human users have different behaviour patterns compared to API clients, and therefore require different treatment. The most vivid distinction comes from the fact that many APIs are very data intensive, designed for batch operations and data dumping, whereas applications for human users are more "reactive" and often do things step-by-step, request-by-request. As a consequence, in many projects APIs URL design is optimised to avoid wasting client and server resources on multiple network roundtrips and repeat storage calls.
Under the hood, API implementations often have different design from core application, optimised for the kind of operations APIs provide. For example, API implementation may use a separate caching strategy. Now if you split the code out, you may want to create a cluster of hosts that only handle the API calls. That is where placing API on another domain becomes beneficial for load management: a separate domain allows for simpler load balancing on high-load sites. In comparison, when you use /api URL prefix on the same domain name (but have separate clusters) then you need a smart (L7-aware) load balancer to do the job of splitting the request flow between API and web front end clusters, but such load balancers are more expensive.
So there may be very good technical reasons why the likes of Twitter separate out the API, but references to other implementations may not apply to YOUR project. If you are at early stages of design, you may want to start with a unified URL scheme on the same domain, but eventually you may find that there are good real-life use cases that make you change the approach, and then ... refactoring.
P.S. there is a lengthy discussion on versioning here - Best practices for API versioning?
P.S.S. I find strongly typed URLs helpful in quick debugging. You can simply put a URL into the browser with .json and quickly get the result without switching to the command line. But agree with you that "accept" header is the preferred method
P.S.S.S. SEO for APIs? I can see how a good URL design can be beneficial, but for a search engine its probably irrelevant if your service provides multiple output formats on the same path / domain name. In the end of the day, search engines are built for human users, and human users don't consume XML and JSON.