Your simple approach is essentially exactly what angular or backbone are doing, so yes you can build a single page app this way. However, your simple example would not be considered a single page app in my opinion. I would consider it simply a single page. A single page application has routes (multiple "pages" with different urls). To accomplish this, you would use javascript's window.location to manipulate the url when you change partials. You would have to handle nested routes and the case when the user goes directly to a subpage (like /contact) in their browser.
There are a lot of tricky edge cases to handle and it would probably be pretty time consuming to implement this on your own, so why not use a framework that was designed exactly for this? Rails is an awesome web development framework, but it is not designed for single page apps. Rails was designed to render html pages server side and a large portion of its code does just that. Building a single page app in rails means carrying a lot of dead weight. All you really need server side for a single page app is a JSON API and a static file server (like NGINX).
Moving the task of rendering dynamic html from the server side to the browser means a lot more javascript code. Its really easy to make a mess using plain jQuery. jQuery is a library, not a framework, meaning jQuery does not provide you with any structure, or code reuse patterns to keep things organized. Angular and Backbone are MVC frameworks, like Rails, but for the browser. They provide you code organization that is crucial for a well structured, testable, maintainable application. In purely jQuery applications, you often see code that handles the data model mixed in directly with DOM manipulation (the view). In an MVC framework, this would be considered sinful.
So really the question of whether you should use a front-end framework in building a single page app is the same question as should you use a back-end framework for building a traditional application. Should you use Rails or just plain Ruby? Sure, you could use just plain Ruby, just like you could use just plain jQuery, but why would you? No need to reinvent the wheel.