There are two aspects here: security aspect and "technical" aspect.
Technical aspect is that the certificate KeyUsage and ExtKeyUsage extensions of the certificate are different for server-side and client-side certificate. Worker will inspect the value of those extensions and complain. This will happen unless you implement custom validator on the worker (in which case any certificate you want will work).
Security aspect is that if the private key leaks for whatever reason, having different certificates (and so private keys) increases security to certain extent.