The CSR can be deleted. Whenever your certificate expires, you can generate a new CSR.
The private key will be used by your web server along with the certificate to establish the SSL connection. They are used either separately, or together as part of a keystore...it depends on how your web server is configured. You may also need any intermediate CA certs to form the certificate chain that is passed to the client.
You definitely want to keep the private key in a secure location so no one can get to it.