It's more than not recommended, If you store or directly handle card holder data yourself you must go through the lengthy and expensive process of becoming PCI compliant, failure to do so can result in you being held liable for penalties and losses by the card schemes in the event of a breach of your system.
A service like Stripe helps you largely avoid this burden.
... but how to make a refund if I don't have any credit card info available in my database?
Using Stripe's API or Dashboard all you need to refund is the Order Id generated by their system when the charge was initially made. You should never need to see, or even be able to see the actual card number.