This is one of the reasons why JSP(X) is abandoned, deprecated and succeeded by Facelets 4 years ago. JSP(X) doesn't offer easy templating capabilities in order to achieve this kind of requirements.
Just migrate to Facelets. Rename .jspx
to .xhtml
and replace <jsp:xxx>
tags by <ui:xxx>
tags. Your functional requirement is answered in 2nd example of How to include another XHTML in XHTML using JSF 2.0 Facelets?
Facelets is natively available in JSF 2.x. If you're still using JSF 1.x, then you can just install it separately as per its documentation.