Using both an ORM and JDBC in the same application is a reasonable thing to do. Hibernate can run native SQL queries so you might consider that as an alternative depending on what you have in mind.
JDBC doesn't use a session factory or entity manager factory comparable to Hibernate. Caching for the JDBC results would have to be handled separately from Hibernate.
The Hibernate/JPA annotations will be irrelevant to JDBC queries, the RowMapper passed into the query controls how pojos get populated. You could re-use your Hibernate entities to be populated by the rowmappers but in practice the reason you use JDBC is to get more focused results with flatter objects and you end up with a separate set of DTOs.
I'm not clear on what you think is going to hurt performance here. JDBC should be an improvement over Hibernate performance-wise, though at the cost of having business-logic spread out in multiple places. The tricky part will probably be setting up the transaction manager so that JDBC and Hibernate use the same transaction, see this question.